Processes, signals and systemd

SIGTERM is a request to terminate. The process can catch it and shut down gracefully: stop accepting new work, finish in-flight requests, flush buffers, close database connections and exit. It is the default signal of kill and what systemd, Docker and Kubernetes send first. SIGKILL cannot be caught, blocked or ignored; the kernel ends the process immediately, so no cleanup happens and you may get corrupted files, stale locks or lost jobs. The right procedure is TERM, wait a reasonable timeout, then KILL only if the process is still alive. That is exactly what systemd does with TimeoutStopSec.

I would not start it by hand over SSH. The simplest robust option is a systemd unit: run it as a dedicated user, set WorkingDirectory and an EnvironmentFile for secrets, ExecStart with the node binary, Restart=on-failure, and enable it so it starts at boot. Logs go to the journal. Alternatives are pm2, which adds clustering and its own log handling and is started at boot through a systemd unit generated by pm2 startup, or running it in a container with a restart policy. In all cases I bind the app to localhost, put nginx in front for TLS and handle SIGTERM in code for zero-surprise restarts.

I start with uptime and compare the load average to nproc. Then top or htop, sorted by CPU and then memory, to find the heavy process. free -h shows whether memory is exhausted and swap is in use, and journalctl -k or dmesg shows whether the OOM killer has been killing processes. If CPU is low but load is high, processes are probably waiting on disk, so I check iostat or the wa value in top and df -h for a full disk. Finally I look at the application side: slow queries, a stuck queue worker, or a traffic spike visible in the nginx access log.

restart stops the service and starts it again, so there is a brief moment with no process and existing connections are dropped. reload asks the running process to re-read its configuration, usually by sending SIGHUP or running a configured ExecReload command, and keeps serving. nginx, for example, starts new worker processes with the new config and lets old ones finish their requests, so a reload is invisible to users. Not every service supports reload; systemctl reload-or-restart handles both cases. Before reloading nginx I always run nginx -t, because a broken config fails the reload and you want to catch that first.

For a small, simple job, cron is perfectly fine and everyone knows its syntax. For important jobs I prefer systemd timers. The job runs as a normal service, so its output goes to the journal with timestamps, it can have resource limits and a dedicated user, it will not start a second copy while the first is still running, and Persistent=true catches up on runs missed while the server was down. systemctl list-timers also shows when each job last ran and runs next, which cron cannot. The cost is two small files instead of one line.

systemctl status name shows the state, the exit code and the last log lines, which is often enough. For more detail, journalctl -u name -b shows all its output since boot, and adding -p err filters to errors. Common causes are a wrong path in ExecStart, a missing environment variable, a port already in use (check with ss -ltnp), file permissions for the service user, or a syntax error in the unit itself, which systemd-analyze verify can catch. If I edited the unit, I make sure I ran daemon-reload. Once fixed, systemctl reset-failed clears the failed state.

PID 1 has special duties. The kernel does not apply default signal handlers to it, so a SIGTERM that the app does not explicitly handle is simply ignored, and the container waits for the timeout before being killed. PID 1 must also reap orphaned children; an app that spawns subprocesses, such as a shell wrapper or headless browser, accumulates zombies if it never calls wait. On a host, systemd is PID 1 and handles both. In containers, either use a tiny init such as tini (docker run --init) or make sure the app handles signals and does not spawn unmanaged children. Also start the app with exec form in the Dockerfile or with exec in the entry script, so it receives signals instead of an intermediate shell.

All of them restart processes; the differences are scope and dependencies. A plain systemd unit needs nothing extra, gives cgroup limits, sandboxing and journal logs, and is the most robust choice for one or two services per host. pm2 adds Node-specific comfort, such as cluster mode across cores, zero-downtime reloads and a deploy workflow, at the cost of another daemon with its own log files. supervisor is simple and language-neutral, popular for running several queue worker processes, but has no cgroup limits of its own. Containers bring reproducible images and isolation, which pays off with many services or teams. Whichever you choose, avoid stacking managers without reason, and be clear which layer owns restarts, logs and limits.

Assume something will leak and decide in advance who should die. Put each service in its own cgroup with MemoryHigh as a soft limit, so the kernel reclaims and slows the service first, and MemoryMax as a hard limit, so the OOM killer acts inside that cgroup only. Lower the OOM score of critical services such as the database with OOMScoreAdjust. Let the restart policy recover the leaking service and alert on NRestarts or OOM events, so the leak is visible instead of silently healed. Keep some swap to absorb spikes, and consider systemd-oomd, which can act earlier on memory pressure. The trade-off is that limits set too low cause restarts under legitimate peaks, so base them on observed usage plus headroom.