Linux Processes, Signals, Jobs and Scheduling

Anil Jalela · Published Sep 3, 2026 · 8 min read

A process is a running execution context with identity, ancestry, open resources, namespaces, limits and state. A process name alone is not a safe target. Before sending a signal or changing priority, identify the exact PID, service unit, owner, workload purpose and dependency. Signals request behavior; they do not universally mean restart, reload or termination. The receiving program and its documented handlers determine the result.

Identify ownership before controlling execution

Shell jobs belong to an interactive shell and use job identifiers such as %1. System processes use PIDs and are often owned by systemd units. Containers and namespaces can present a different PID view. Use systemctl for a managed service because the unit expresses restart, timeout, cgroup and dependency policy that a raw kill bypasses.

A zombie has completed but its parent has not collected the exit status. Sending signals to the zombie cannot make it exit again. Inspect the parent and application behavior. An orphaned process is reparented; it is not automatically a zombie or an error.

Read a process as an owned execution tree

Control a process without guessing

1. Capture identity and start time. A reused PID can otherwise target a different process.
2. Map the owner. Connect the PID to its systemd unit, container, user session or approved batch job.
3. Preserve the symptom. Capture logs, open files, sockets, stack or application state before changing execution.
4. Choose the least disruptive request. Use an application command, unit reload or SIGTERM before SIGKILL when supported.
5. Wait for the defined timeout. Watch child processes, queue drain and file consistency.
6. Verify cleanup and service outcome. Confirm the expected PID, sockets, locks, jobs and downstream behavior.

Inspect ancestry, state and unit ownership

ps -eo pid,ppid,user,lstart,stat,ni,pcpu,pmem,wchan:24,comm,args --sort=-pcpu
pstree -aps
systemctl status example.service
systemctl show example.service -p MainPID -p ControlGroup -p Restart
cat /proc/1234/status
ls -l /proc/1234/fd
ss -lntup
renice 5 -p 1234
kill -TERM 1234
systemctl kill --kill-who=main --signal=TERM example.service
systemctl list-timers --all
crontab -l

• Replace PID 1234 and the unit name only after resolving them from current evidence.
• STAT=Z identifies a zombie; inspect PPID and the parent rather than sending SIGKILL to the zombie.
• kill -TERM sends a signal and does not prove termination. Observe state, logs and application cleanup.
• Prefer systemd timers for service-owned scheduling that needs dependency, missed-run, logging and unit-policy behavior.

Evidence and acceptance criteria

Worked scenario: repeated SIGKILL causes a longer outage

A Java service stops answering health checks. An operator repeatedly uses SIGKILL because it is immediate. Each kill prevents graceful connection drain and leaves recovery work to the next start; dependent jobs retry and increase load. The actual symptom is a blocked storage call visible in process state and storage latency.

The corrected runbook captures thread and I/O evidence, stops new traffic, asks systemd for a controlled stop, waits for the approved deadline and escalates only if the process cannot exit. Storage ownership is repaired before a bounded restart. Acceptance includes queue drain and downstream latency, not just a new PID.

Practical how-to cases

ps -eo pid,ppid,user,stat,ni,pcpu,pmem,comm,args --sort=-pcpu | head
pstree -aps
systemctl status 1234 2>/dev/null || true
cat /proc/1234/status
ls -l /proc/1234/fd | head

kill -TERM 1234
for n in 1 2 3 4 5; do ps -p 1234 -o pid,stat,cmd; sleep 1; done
journalctl _PID=1234 --since '-5 minutes' --no-pager

systemd-analyze calendar 'Mon..Fri 02:15'
systemctl cat report.service report.timer
systemctl enable --now report.timer
systemctl list-timers --all | grep report
systemctl start report.service
journalctl -u report.service -n 20

sleep 300 &
jobs -l
kill -STOP %1
jobs -l
kill -CONT %1
fg %1

echo 'logger -t at-lab one-time' | at now + 2 minutes
atq
crontab -l
systemctl list-timers --all
journalctl -t at-lab --since '-10 minutes'

Independent practice tasks

1. Diagnose a CPU-bound and an I/O-wait lab process.
2. Convert one cron entry to a systemd timer with logs.
3. Apply a nice adjustment and prove it does not impose a hard CPU limit.
4. Recover a failed scheduled job caused by a missing environment variable.

For this lesson on Linux Processes and Scheduling, complete each task without copying the worked command sequence. Record the initial state, exact change, verification, negative test and recovery command. A task is unfinished if it works now but does not survive a reboot where persistence is required.

Troubleshooting by symptom

Unsafe operations and recovery boundaries

• Unsafe: kill -9 prevents application cleanup and should be a documented final escalation after evidence and graceful options.
• Unsafe: killall or name-based broad matching can terminate unrelated tenants or versions. Resolve exact PIDs and units.
• Unsafe: running durable business jobs with nohup alone omits service ownership, restart policy, limits and reliable evidence.

Rewritten knowledge checks

Guided lab and acceptance test

1. Start a harmless sleep process, record PID/start time/parent and move it between foreground and background.
2. Send STOP, CONT and TERM while observing process state and exit status.
3. Create a transient systemd unit with a memory limit and inspect its cgroup ownership.
4. Create a timer that writes a UTC timestamp to a protected lab log and verify last/next run.
5. Create a short-lived child whose parent delays waiting; observe states without attempting to kill a zombie.
6. Remove the lab units and prove no process, timer or output path remains.

Primary references

• RHEL 10 configuring basic system settings
• RHEL 10 documentation
• RHEL 9 documentation