Linux log management: Using journalctl commands for systemd logs

Level: Beginner
Estimated time: ~20 minutes
Goal: Learn how to query, filter, and manage system and service logs, utilizing Linux troubleshooting logs to resolve issues and maintain server uptime.

Introduction

Reliable Linux log management is essential for maintaining server uptime and performing Linux server diagnostics. When managing web servers, you need to quickly analyze system logs with journalctl to find the root cause of an issue. The Systemd (the init system and service manager for most modern Linux distributions) collects system logs (general records of system operations and system-wide events). A single Log (a record of events that happen within an operating system or software application) is collected by Systemd-journald (a system service that collects and stores logging data) and stored in the Journal (the binary log data generated and managed by systemd-journald). To view this data, we use Journalctl (a command-line utility used to query and display logs from systemd). This tutorial focuses on using journalctl commands to query, filter, and manage journalctl systemd logs. By the end, you will confidently navigate the command line and use Linux troubleshooting logs to resolve issues efficiently.

Prerequisites

Before you begin, make sure the following conditions are met:

• Operating system: Ubuntu 20.04/22.04/24.04 LTS, Debian 11/12/13, or AlmaLinux/Rocky 8/9/10
• Hardware & Network requirements: None (this guide uses basic command-line utilities)
• Access: sudo or root access to the server
• Required knowledge: basic Linux command line usage

Step 1: Linux log management: Understanding volatile and persistent logs

By default, some systems configure the journal to use volatile logs (logs stored only in RAM and lost upon reboot). For effective Linux log management, we need persistent logs (logs saved to disk across reboots) so you can investigate crashes after a restart.

Run the following command to check your storage configuration:

sudo grep -i 'storage' /etc/systemd/journald.conf

Expected output:

#Storage=auto

You should see output indicating whether the storage is set to auto, persistent, or volatile. A commented-out #Storage=auto means it uses the default auto behavior.

On Ubuntu 24.04 and Debian 13, the /var/log/journal directory already exists, and persistent logging works out of the box. If the directory is missing (such as on a fresh AlmaLinux installation) and you want to enforce persistent logging, create the directory and restart the systemd logging system (the systemd-journald service manages this directory):

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

For AlmaLinux/RHEL systems, you must also flush the logs from the volatile /run/log/journal location to the new persistent disk storage:

sudo journalctl --flush

You should see no output. This confirms the service restarted successfully and will now write persistent data to the disk.

Step 2: Basic journalctl commands for systemd logs

To view all available entries, use the default command.

Run the following command:

sudo journalctl

Expected output:

May 20 10:00:00 server systemd[1]: Started Logging Service.
May 20 10:00:01 server kernel: Linux version 5.15.0-101-generic...

You should see a paginated list of all systemd logs on your server. Press q to exit the pager.

To analyze logs with journalctl effectively, you rarely want to read everything from the beginning. You can reverse the output to see the newest entries first using the -r flag.

Run the following command:

sudo journalctl -r

Expected output:

May 23 20:00:00 server sshd[1234]: Accepted publickey for user from 192.168.1.50...
May 23 19:59:58 server systemd[1]: Session 4 created for user.

You should see the most recent log entries at the top of your screen. This is a crucial technique to analyze logs with journalctl quickly after an incident.

Step 3: How to check service logs

Often, you need to troubleshoot a specific Unit (an object that systemd knows how to manage), such as a service unit (a specific type of unit that controls a service, like nginx or sshd). Use the -u flag to check service logs for specific services like the nginx web server.

Run the following command (assuming you have installed the service, e.g., via sudo apt install nginx -y or sudo dnf install nginx -y):

sudo journalctl -u nginx

Expected output:

May 23 18:00:00 server systemd[1]: Starting A high performance web server and a reverse proxy server...
May 23 18:00:01 server systemd[1]: Started A high performance web server and a reverse proxy server.

You should see only the log entries generated by the nginx service. This helps isolate web traffic errors from other background system events. If the service is not installed, you will simply see -- No entries --.

To check service logs for the SSH daemon, the unit name varies by distribution.

For Ubuntu/Debian, run:

sudo journalctl -u ssh

For AlmaLinux/RHEL/Rocky, run:

sudo journalctl -u sshd

Expected output:

May 23 19:50:00 server sshd[1234]: Invalid user admin from 10.0.0.5 port 55432
May 23 19:55:00 server sshd[1235]: Accepted publickey for root from 10.0.0.2 port 44322

You should see authentication attempts and SSH daemon events.

Step 4: journalctl filtering logs by time and type

When dealing with large amounts of data, you need log filtering (the process of narrowing down log output based on specific criteria). journalctl filtering logs by time is incredibly useful when an outage occurred at a known time.

To see messages generated since the last boot, we check the boot logs (records of events that occur during the system startup process).

Run the following command:

sudo journalctl -b

Expected output:

May 23 08:00:00 server kernel: Linux version 5.15.0-101-generic...
May 23 08:00:00 server kernel: Command line: BOOT_IMAGE=/boot/vmlinuz...

You should see output starting from the very first event of the current boot sequence.

For time-based journalctl filtering logs, use --since and --until.

Run the following command:

sudo journalctl --since "1 hour ago"

Expected output:

May 23 19:00:00 server cron[567]: (root) CMD (/usr/local/bin/backup.sh)
May 23 19:05:00 server sudo[580]: user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/bin/ls

You should see all events that occurred in the last 60 minutes.

To view kernel logs (messages generated by the Linux kernel, typically hardware and driver events), use the -k flag.

Run the following command:

sudo journalctl -k

Expected output:

May 23 08:00:00 server kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex
May 23 08:00:01 server kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

You should see low-level kernel messages, useful for diagnosing hardware or driver issues.

Step 5: Filtering by Priority

Every entry has a Priority (the severity level assigned to a log message). The levels range from Debug (the lowest priority level, used for detailed troubleshooting information) to Error (a priority level indicating a failure in a service or process). A commonly checked level is Warning (a priority level indicating potential issues that are not currently errors).

For journalctl filtering logs by priority, use the -p flag.

Run the following command to see only errors:

sudo journalctl -p err

Expected output:

May 23 10:15:20 server systemd[1]: Failed to start Custom Application Service.
May 23 14:30:00 server sshd[1122]: error: kex_exchange_identification: Connection closed by remote host

You should see a condensed list containing only error-level messages, filtering out normal operations.

Run the following command to see warnings and errors:

sudo journalctl -p warning

Expected output:

May 23 10:15:15 server systemd-udevd[330]: vda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/vda' failed with exit code 1.
May 23 10:15:20 server dhcpcd[400]: eth0: no IPv6 Routers available

You should see both warnings and errors, giving you a broader view of potential problems. On a clean operating system, it is common to see routine warning messages from services like multipathd, irqbalance, dhcpcd, or udev, rather than critical service failures. The exact output heavily depends on your specific system configuration and environment.

Step 6: How to monitor Linux logs in real time

When applying configuration changes or reproducing an issue, it is best to monitor Linux logs in real time. Use the -f (follow) flag.

Run the following command:

sudo journalctl -f

Expected output:

May 23 20:05:00 server sudo[2001]: user : TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=/bin/bash
May 23 20:05:00 server su[2002]: (to root) user on pts/1
May 23 20:05:00 server su[2002]: pam_unix(su:session): session opened for user root by user(uid=1000)

You should see the latest log entries, and the terminal will hang, printing new events as they occur. Press Ctrl+C to exit.

To monitor Linux logs in real time for the SSH daemon, the unit name varies by distribution.

For Ubuntu/Debian, run:

sudo journalctl -u ssh -f

For AlmaLinux/RHEL/Rocky, run:

sudo journalctl -u sshd -f

Expected output:

May 23 20:10:15 server sshd[2100]: Connection from 192.168.1.100 port 45678
May 23 20:10:17 server sshd[2100]: Accepted publickey for admin from 192.168.1.100 port 45678
May 23 20:10:17 server sshd[2100]: pam_unix(sshd:session): session opened for user admin by (uid=0)

You should see live SSH authentication attempts as they happen.

Step 7: Disk space and log rotation

The systemd journal can grow large over time. The practice of archiving and deleting old logs to save space is called log rotation. You can check how much space the systemd journal currently uses.

Run the following command:

sudo journalctl --disk-usage

Expected output:

Archived and active journals take up 200.0M in the file system.

You should see an output indicating the total space used by your log files.

To perform manual log rotation and free up disk space, you can vacuum the data by time or size.

Run the following command to keep only the last 500MB of data:

sudo journalctl --vacuum-size=500M

Expected output:

Vacuuming done, freed 0B of archived journals from /var/log/journal.

You should see output indicating that old files have been deleted until the total size is under 500MB.

Verification

To ensure your logging setup is working correctly and persistent storage is active:

1. Verify the log storage location has been created:

   ls -ld /var/log/journal

   You should see a directory owned by root. Depending on your distribution, the group might be systemd-journal or root (both are normal).

2. Confirm journald is actively running:

   sudo systemctl status systemd-journald

   You should see the service is active (running).

Troubleshooting

If you encounter issues while managing logs, check these common scenarios:

• Empty service logs (-- No entries --): Ensure the service is actually installed and running. Also, verify you are using the correct unit name for your distribution (e.g., ssh on Debian/Ubuntu vs. sshd on RHEL/AlmaLinux).
• Missing /var/log/journal: On some systems (like AlmaLinux), you must manually create this directory and restart the logging service to enable persistent logs. If you skip this step, logs will remain in volatile memory (/run/log/journal).
• Permission denied: If you cannot read the logs without sudo, ensure your user is part of the systemd-journal or adm group (sudo usermod -aG systemd-journal $USER).

Reverting changes

If you want to disable persistent logging and return to volatile logs (to save disk space, for instance):

1. Remove the persistent storage directory:

   sudo rm -rf /var/log/journal

2. Restart the logging service to recreate the volatile storage in /run/log/journal:

   sudo systemctl restart systemd-journald

Conclusion

That is all. You now understand the essentials of Linux log management. Using the right journalctl commands, you can efficiently navigate systemd logs, apply filters, and manage disk space. Whether you need to analyze logs with journalctl after a crash or just check routine service health, mastering journalctl is a critical step in maintaining a healthy infrastructure.

Document Version: 1.0
Last Updated: May 2026
Owner: Technical Documentation Team