RHEL Logging with journald, rsyslog and logrotate

Anil Jalela · Published Sep 3, 2026 · 7 min read

Logs are evidence only when their source, time, completeness, retention and access are understood. /var/log/messages is not a universal answer: systemd services write to the journal, applications may use files or structured remote telemetry, and rsyslog can route selected events. Rotation controls local files but does not guarantee durable central retention or successful ingestion.

Design the evidence path before an incident

Record which layer emits an event, how journald receives and stores it, whether rsyslog forwards or writes it, where central storage timestamps it and how long each copy remains. Preserve event time and ingestion time when delayed delivery matters. Synchronize clocks, but never assume all sources use the same timezone or precision.

Logs may contain addresses, identifiers, message content, tokens or secrets. Collection needs least privilege, transport protection, retention and deletion policy. Debug logging can expose more sensitive data and consume storage rapidly.

Follow an event from producer to retention

Build and test a logging contract

1. List critical services, event classes, owners, retention and alert use.
2. Confirm UTC/time synchronization and record boot IDs.
3. Set journal storage and limits appropriate to local recovery needs.
4. Validate rsyslog syntax and configure protected forwarding with queueing where required.
5. Configure logrotate only for application files not managed by another owner.
6. Emit a unique harmless test event and trace it through local and remote destinations.
7. Test rotation, service reopen behavior, storage pressure and remote outage recovery.

Query journals and validate logging components

timedatectl
chronyc tracking
journalctl --list-boots
journalctl -u sshd.service -b --since '-30 minutes' --no-pager
journalctl -p warning..alert -b --no-pager
journalctl --disk-usage
journalctl --verify
systemd-analyze cat-config systemd/journald.conf
rsyslogd -N1
logrotate --debug /etc/logrotate.conf
logger --tag nitwings-lab --priority user.notice 'logging path verification'
journalctl -t nitwings-lab --since '-5 minutes' --no-pager

• journalctl --verify checks journal-file consistency, not whether every expected producer emitted or forwarded events.
• A successful rsyslog syntax check does not prove TLS trust, network delivery or remote indexing.
• Use a unique benign test marker and remove no evidence during an active incident.
• Do not force rotation against active application logs until reopen/copytruncate behavior and data-loss boundary are understood.

Evidence and acceptance criteria

Worked scenario: disk fills although logs were rotated

A large application log is renamed and compressed, but the process keeps the old file descriptor open. The pathname looks small while the deleted inode still consumes space. Repeated deletion does not release it.

The operator proves the deleted-open descriptor with lsof +L1, confirms the application supports a documented reopen signal or controlled reload, and watches the descriptor move to the new file. Rotation is changed to use the application’s supported reopen behavior. Disk, journal and remote evidence remain intact.

Practical how-to cases

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --flush
journalctl --list-boots
journalctl --disk-usage

timedatectl
journalctl --list-boots
journalctl -u sshd -b --since '2026-09-04 10:00:00 UTC' --until '2026-09-04 10:15:00 UTC' -o short-iso-precise
journalctl _PID=1234 --no-pager

sudo rsyslogd -N1
logger -p local0.notice -t nitwings-lab 'forwarding check'
journalctl -t nitwings-lab --since '-5 minutes'
sudo ss -ntup | grep 6514 || true
sudo journalctl -u rsyslog -n 30

sudo logrotate --debug /etc/logrotate.conf
sudo logrotate --force /etc/logrotate.d/labapp
ls -l /var/log/labapp*
sudo lsof /var/log/labapp.log
journalctl -u labapp -n 20

Independent practice tasks

1. Set journal size and retention limits for a small lab disk.
2. Route one facility to a dedicated file with secure permissions.
3. Configure a TLS forwarding queue and simulate collector outage.
4. Recover a service that keeps writing to a rotated deleted file.

For this lesson on RHEL Logging and Retention, complete each task without copying the worked command sequence. Record the initial state, exact change, verification, negative test and recovery command. A task is unfinished if it works now but does not survive a reboot where persistence is required.

Troubleshooting by symptom

Unsafe operations and recovery boundaries

• Unsafe: deleting or vacuuming logs during an incident destroys evidence and may not free deleted-open space.
• Unsafe: enabling verbose debug globally can expose secrets and exhaust storage; scope and time-bound it.
• Unsafe: forwarding logs without authenticated encryption or access controls can disclose operational and customer data.

Rewritten knowledge checks

Guided lab and acceptance test

1. Record time state and boot ID, then query one unit by time and priority.
2. Configure a lab rsyslog file action for a unique facility/tag and validate syntax.
3. Emit a unique logger event and trace fields through journal and file.
4. Create a logrotate rule for the lab file and run debug before a forced lab-only rotation.
5. Simulate a held-open file and use lsof +L1 to explain space.
6. Remove the lab routing/rotation files, reload safely and verify ordinary logging continues.

Primary references

• RHEL 10 configuring basic system settings
• RHEL 10 documentation
• RHEL 9 documentation