Packages, logs, cron and disks
Keep the server updated, know where the logs are, schedule jobs and never run out of disk.
In 30 seconds
Most server maintenance comes down to four things: installing and updating software with the package manager, reading and rotating logs, scheduling recurring jobs, and watching disk and memory. A full disk alone can take down a database, a web server and a deploy at once. This lesson gives you the commands and habits to keep a server healthy.
Why I use it
- unattended-upgrades is part of my baseline for every server, because unpatched packages are one of the easiest ways in.
- I cleaned one production server from 91% to 66% disk usage by tracking down old logs, old builds and oversized node_modules.
- Disk, load and uptime monitoring is on my standard list, because a full disk is a quiet failure until everything breaks at once.
How it works
01apt: installing and updating software
On Debian and Ubuntu, apt installs software from trusted repositories, resolves dependencies and verifies package signatures. apt update refreshes the list of available versions; it does not install anything. apt upgrade installs newer versions of what you already have, and apt full-upgrade may also add or remove packages when dependencies change, which is needed for kernel updates. On RHEL-family systems the same jobs are done with dnf (dnf check-update, dnf upgrade). Prefer packages from the distro repositories; add third-party repositories only from vendors you trust, using their signed-by keyrings. After a kernel or core library update, check whether a reboot is required.
sudo apt update # refresh package lists
apt list --upgradable # what would change
sudo apt upgrade # install updates
sudo apt full-upgrade # also handle changed dependencies
sudo apt install nginx htop # install packages
sudo apt remove --purge apache2 # remove, including config
sudo apt autoremove # remove unused dependencies
apt show nginx; apt policy nginx # details and versions
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required
02Automatic security updates
unattended-upgrades installs security updates automatically every day, so known holes get patched even when nobody logs in. It is installed by default on Ubuntu Server; you can check and enable it with dpkg-reconfigure. By default it only applies updates from the security pocket, which is a sensible balance between safety and stability. You can configure it to reboot automatically at a quiet hour when a kernel update needs it, and to clean up old kernels. On RHEL the equivalent is dnf-automatic. Its log, in /var/log/unattended-upgrades/, shows what was installed and when. Automatic updates are not a replacement for planned upgrades between major versions, which you should test first.
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades # enable
cat /etc/apt/apt.conf.d/20auto-upgrades
# In /etc/apt/apt.conf.d/50unattended-upgrades you can set:
# Unattended-Upgrade::Automatic-Reboot "true";
# Unattended-Upgrade::Automatic-Reboot-Time "04:00";
# Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
sudo unattended-upgrade --dry-run --debug # test it
03Logs: /var/log, journald and logrotate
There are two log systems on a modern server. systemd's journal collects output from services and the kernel; you read it with journalctl. Classic text files in /var/log come from rsyslog and from applications such as nginx. Useful files on Ubuntu are /var/log/syslog for general messages, /var/log/auth.log for logins and sudo, and /var/log/nginx/ for web traffic and errors (RHEL uses /var/log/messages and /var/log/secure). Logs grow forever unless something trims them. logrotate does that for text files: it renames the current file, compresses old ones and keeps only a set number. For the journal, set a size cap in journald's config or clean it with vacuum.
# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
copytruncate
}
# Test without changing anything: sudo logrotate -d /etc/logrotate.d/myapp
# Journal size: journalctl --disk-usage
# sudo journalctl --vacuum-size=500M
# Permanent cap: SystemMaxUse=500M in /etc/systemd/journald.conf
04Scheduling with cron
cron runs commands on a schedule. Each user has a crontab (edit it with crontab -e), and system jobs live in /etc/crontab and /etc/cron.d. A line has five time fields, minute, hour, day of month, month and day of week, followed by the command. An asterisk means every, */5 means every five, and ranges like 1-5 mean Monday to Friday. Cron runs with a minimal environment: PATH is short and your shell profile is not loaded, so use absolute paths and redirect output to a log, otherwise errors vanish. Cron uses the server's time zone, which is often UTC on cloud servers. A site like crontab.guru helps you check an expression.
# m h dom mon dow command
*/5 * * * * /usr/local/bin/healthcheck.sh >> /var/log/healthcheck.log 2>&1
0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
30 2 * * 1 /usr/local/bin/weekly-report.sh
# Laravel scheduler: run every minute as the app user
* * * * * cd /var/www/app && php artisan schedule:run >> /dev/null 2>&1
# Prevent overlapping runs with flock
0 * * * * flock -n /tmp/sync.lock /usr/local/bin/sync.sh
05Disks: df, du, lsblk and mount
df shows how full each mounted filesystem is; du shows how much space files and directories use. lsblk lists block devices, so you can see disks and partitions, including a new volume you just attached. To use a new volume, create a filesystem on it once (ext4 is a safe default), mount it, and add it to /etc/fstab by UUID so it comes back after a reboot. Test fstab with mount -a before rebooting, because a broken entry can stop the server from booting. Also watch inodes with df -i: a disk can be 'full' with free space left if millions of tiny files (sessions, cache) use up all inodes.
df -h # space per filesystem
df -i # inodes per filesystem
lsblk -f # disks, partitions, filesystems, UUIDs
sudo mkfs.ext4 /dev/sdb # format a NEW empty volume (destroys data!)
sudo mkdir -p /mnt/data
sudo mount /dev/sdb /mnt/data
# /etc/fstab line (use the UUID from lsblk -f):
# UUID=your-uuid /mnt/data ext4 defaults,nofail 0 2
sudo mount -a # test fstab before rebooting
06Finding what fills a disk
When df shows a disk at 90% or more, work from the top down. du with a depth limit shows the biggest directories; repeat inside the largest one. ncdu is an interactive version that makes this fast. The usual suspects are old logs and rotated archives, the systemd journal, Docker images and build cache, old release folders from deployments, node_modules and vendor directories, database dumps forgotten in home directories, and apt's package cache. One trap: if you delete a log file that a process still has open, the space is not freed until that process restarts or reopens the file. lsof +L1 lists those deleted-but-open files. Truncating a live log is safer than deleting it.
sudo du -xh --max-depth=1 / | sort -h | tail
sudo du -xh --max-depth=1 /var | sort -h | tail
sudo ncdu -x / # interactive (apt install ncdu)
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
journalctl --disk-usage
docker system df # images, containers, cache
sudo apt clean # apt package cache
sudo lsof +L1 # deleted files still held open
sudo truncate -s 0 /var/log/myapp/huge.log # empty a live log safely
07Memory and swap
free -h shows memory. Do not panic at a small 'free' number: Linux uses spare memory as a disk cache and gives it back when apps need it. The column to read is available. When available memory runs out, the kernel's OOM (out of memory) killer ends a process, often the biggest one, which is frequently your database. Swap is disk space used as overflow memory. It is much slower than RAM, but on a small VPS a modest swap file (1 to 2 GB) turns a sudden crash into a slowdown you can notice and fix. If the server swaps all the time, you need more RAM or a less hungry app, not more swap.
free -h # read the "available" column
sudo journalctl -k | grep -i "out of memory" # OOM kills
# Create a 2 GB swap file
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo "/swapfile none swap sw 0 0" | sudo tee -a /etc/fstab
swapon --show
Use it when
- Every server you run: updates, log rotation and disk monitoring are basic hygiene.
- Disk-full incidents, failing deploys or databases refusing writes.
- Recurring tasks such as backups, reports, cleanups and framework schedulers.
- Small VPS instances where a swap file and journal limits prevent avoidable crashes.
Skip it when
- Do not run apt full-upgrade or major release upgrades on production without a snapshot and a maintenance window.
- Do not use swap as a substitute for enough RAM on a busy database server.
- Do not delete files you do not understand just to free space; check what writes them first.
Alternatives
- dnf / dnf-automatic
The RHEL, Rocky and Alma equivalents of apt and unattended-upgrades.
- systemd timers
A more observable alternative to cron, with journal logs and catch-up runs.
- Central logging (Loki, ELK, hosted)
Ship logs off the server so you can search across machines and keep them after a disk dies.
- Object storage
Move user uploads and backups to S3-compatible storage so the server disk stays small and replaceable.
Best practices
- Enable unattended-upgrades for security updates and check for required reboots.
- Take a snapshot before big upgrades, and upgrade staging first.
- Give every application log a logrotate rule and cap the journal size.
- Alert on disk usage well before it is full, for example at 80%.
- In cron, use absolute paths, redirect output to a log and prevent overlaps with flock.
- Mount extra volumes by UUID with the nofail option, and test fstab with mount -a.
- Keep releases, Docker images and build caches pruned as part of your deploy process.
Common mistakes
- Running apt upgrade without apt update first, so nothing new is installed.
- Deleting a live log with rm and seeing no space freed because the process still holds it open.
- Cron jobs that work in the terminal but fail silently because of PATH or a missing redirect.
- A typo in /etc/fstab that stops the server from booting.
- Reading 'free' instead of 'available' memory and adding RAM that was not needed.
Deep dive
08Under apt: dpkg, sources, pinning and phased updates
apt is the planner; dpkg does the installing. dpkg unpacks .deb archives, runs their maintainer scripts and records state in /var/lib/dpkg. If an upgrade is interrupted, packages stay half-configured until you run dpkg --configure -a. Since Ubuntu 24.04, repositories are described in the deb822 format, in files such as /etc/apt/sources.list.d/ubuntu.sources, and third-party repositories should name their key with Signed-By so that key cannot vouch for other repositories. Pinning in /etc/apt/preferences.d sets priorities between repositories, and apt-mark hold freezes a package version when you must wait before upgrading it. Ubuntu also uses phased updates: non-security updates roll out to a growing percentage of machines over several days, so apt may report some upgrades as 'deferred due to phasing'. That is expected, not a broken mirror.
# /etc/apt/sources.list.d/vendor.sources (deb822 format)
Types: deb
URIs: https://packages.example.com/apt
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/vendor.gpg
# Useful commands:
# sudo dpkg --configure -a # finish interrupted installs
# sudo apt-mark hold nginx # freeze a version (unhold to undo)
# apt-cache policy nginx # which repo and priority wins
# dpkg -S /usr/sbin/nginx # which package owns a file
# zgrep ' install ' /var/log/dpkg.log* # install history
09Filesystems: reserved blocks, growing volumes and LVM
ext4 reserves 5% of blocks for root by default, so services running as root keep working when users fill the disk. On a 500 GB data volume that is 25 GB; tune2fs -m 1 reduces it, but keep the default on the root filesystem. ext4 can be grown online and shrunk only offline; XFS, the default on RHEL, can be grown online but never shrunk. When you enlarge a disk at the provider, the filesystem does not grow by itself: if the disk has partitions, grow the partition with growpart first, then the filesystem with resize2fs (ext4) or xfs_growfs (XFS). Many cloud images do this automatically for the root disk at boot through cloud-init, but attached volumes usually need it by hand. LVM adds a layer that lets you span disks, grow logical volumes and take snapshots, at the cost of more moving parts.
lsblk # disk vs partition sizes
sudo growpart /dev/vda 1 # grow partition 1 to fill the disk
sudo resize2fs /dev/vda1 # grow ext4 online
sudo xfs_growfs / # grow XFS (by mount point)
sudo resize2fs /dev/sdb # unpartitioned volume: grow directly
sudo tune2fs -l /dev/sdb | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sdb # 1% reserved on a data volume
sudo lvextend -r -l +100%FREE /dev/vg0/data # LVM: grow LV and filesystem
10Diagnosing I/O and memory pressure
When a server is slow but CPU is idle, measure the disk. vmstat 1 shows, every second, processes blocked on I/O (b), swap-in and swap-out (si, so) and CPU time waiting for I/O (wa). iostat -x 1 (sysstat package) shows per-device latency in r_await and w_await and utilization; on SSDs and network volumes, latency and the provider's IOPS limit matter more than %util. iotop or pidstat -d 1 shows which process does the I/O. Writes go first to the page cache and are flushed later, so a big write can look fast and then stall everything during writeback. Pressure Stall Information in /proc/pressure/io and /proc/pressure/memory reports the share of time tasks were stalled, which is a far better signal than free memory. vm.swappiness controls how eagerly the kernel swaps; the default of 60 is fine for most servers.
vmstat 1 5 # b, si/so, wa columns
iostat -x 1 3 # r_await, w_await, %util per device
sudo iotop -oPa # processes doing I/O (accumulated)
pidstat -d 1 5 # per-process disk reads/writes
cat /proc/pressure/io # some/full stall percentages
grep -E 'Dirty|Writeback' /proc/meminfo
sysctl vm.swappiness
11Designing the log pipeline
On one server, journald plus logrotate is enough if you cap sizes. journald stores logs in /var/log/journal when that directory exists or Storage=persistent is set; otherwise only in memory under /run, and they vanish at reboot. It also rate-limits noisy services (RateLimitIntervalSec and RateLimitBurst), so a looping error can hide its own repetitions; the journal records how many messages were suppressed. Beyond one server, logs should leave the machine: an agent such as Grafana Alloy (the successor to Promtail), Vector or Fluent Bit tails the journal and files and ships them to Loki, Elasticsearch or a hosted service. Prefer structured JSON logs with consistent fields such as level, request ID and user ID, so you can filter instead of grep. Decide retention per type: debug logs for days, access logs for weeks, audit logs as long as policy requires.
# /etc/systemd/journald.conf.d/size.conf
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=2G
MaxRetentionSec=30day
RateLimitIntervalSec=30s
RateLimitBurst=10000
# Apply: sudo systemctl restart systemd-journald
# Check: journalctl --disk-usage; journalctl --verify
Real-world scenarios
apt is locked or half-finished
Problem
apt install fails with 'Could not get lock /var/lib/dpkg/lock-frontend', and after a dropped SSH session during an upgrade, apt complains that dpkg was interrupted.
How to solve it
First check whether another apt is really running: ps aux | grep -E 'apt|dpkg|unattended' and systemctl status unattended-upgrades apt-daily-upgrade.service. On fresh servers, unattended-upgrades often runs right after boot, so wait for it rather than deleting lock files, which can corrupt the dpkg database. If nothing is running and the previous upgrade was interrupted, run sudo dpkg --configure -a, then sudo apt -f install to fix broken dependencies, then apt update and upgrade again. Read /var/log/apt/term.log for the failing maintainer script. In future, run long upgrades inside tmux so a dropped connection does not kill them.
The provider disk was resized but df shows the old size
Problem
You enlarged an attached volume from 100 GB to 200 GB in the provider's panel. lsblk shows 200 GB, but df -h still reports 100 GB and the disk is still nearly full.
How to solve it
The block device grew, but the partition and filesystem did not. Run lsblk -f to see the layout and filesystem type. If the filesystem sits directly on the device (no partition, common for volumes), grow it online: sudo resize2fs /dev/sdb for ext4 or sudo xfs_growfs /mnt/data for XFS. If there is a partition, first sudo growpart /dev/sdb 1, then resize the partition's filesystem. With LVM, use pvresize and then lvextend -r. Take a snapshot before you start. Then fix the root cause of growth too, because a bigger disk only buys time if logs or uploads grow without limit.
'No space left on device' with gigabytes free
Problem
Uploads and PHP sessions fail with 'No space left on device', yet df -h shows the disk at 60%. Restarting services does not help.
How to solve it
Check inodes with df -i: if IUse% is at 100%, the filesystem has run out of file entries, usually because of millions of tiny files. Find the directory with the most files using sudo find / -xdev -type f | cut -d/ -f2-4 | sort | uniq -c | sort -rn | head. Typical culprits are PHP session files without garbage collection, cache directories or mail queues. Delete old ones with find -mtime +2 -delete in the right directory. Then fix the cause: enable session garbage collection or move sessions to Redis, add a cleanup timer, and alert on inode usage as well as space. For new filesystems that hold huge numbers of small files, choose the inode ratio at mkfs time.
Cheat sheet
- Finish an interrupted package install
sudo dpkg --configure -a && sudo apt -f install- Which package owns a file
dpkg -S /usr/bin/curl- Freeze / unfreeze a package version
sudo apt-mark hold nginx / sudo apt-mark unhold nginx- Grow a partition and ext4 filesystem
sudo growpart /dev/vda 1 && sudo resize2fs /dev/vda1- Directories with the most files (inodes)
sudo du --inodes -x -d 2 / | sort -n | tail- Disk latency per device
iostat -x 1 3- Processes doing I/O
sudo iotop -oPa- Logs from the previous boot
journalctl -b -1 -p warning- Shrink the journal now
sudo journalctl --vacuum-time=14d- Force a logrotate run to test a rule
sudo logrotate -vf /etc/logrotate.d/myapp
In my real projects
- I brought one server down from 91% to 66% disk usage by clearing old logs, old builds and oversized node_modules, and checked its backups afterwards.
- unattended-upgrades and disk, load and uptime monitoring are on the baseline I apply to every server.
- I use cron to run Laravel's scheduler on my servers.
- On the POS SaaS server, logs are collected centrally with Loki and visualized in Grafana next to Prometheus metrics.
Questions & answers
Q1What is the difference between apt update and apt upgrade?
apt update only downloads fresh package lists from the configured repositories, so apt knows which versions exist. It changes no software. apt upgrade then installs newer versions of packages that are already installed, without removing anything. apt full-upgrade goes further and may install new dependencies or remove packages to complete the upgrade, which kernel and some library updates need. So the normal sequence is update, then upgrade. On production I review apt list --upgradable first, take a snapshot before larger changes, and let unattended-upgrades handle routine security fixes daily. Moving between Ubuntu releases is a separate, planned do-release-upgrade.
Q2The disk is full on a production server. What do you do?
First stabilize: df -h to see which filesystem is full, and df -i to rule out inode exhaustion. Then find the cause with du -xh --max-depth=1 sorted by size, drilling into the largest directories, or ncdu. Common culprits are logs without rotation, the journal, Docker images and build cache, old release directories and stray database dumps. I free space safely: truncate live logs instead of deleting them, vacuum the journal, prune Docker, remove old releases. If space does not come back, lsof +L1 shows deleted files still held open by a process. Afterwards I fix the root cause with logrotate rules, journal limits and a disk alert.
Q3Why does my cron job work in the terminal but not from cron?
Cron runs commands with a minimal environment. PATH contains only a few system directories, your .bashrc or .profile is not loaded, the working directory is the user's home, and it uses /bin/sh rather than bash unless you say otherwise. So commands like node or php installed in other places are not found, relative paths break and environment variables are missing. Fix it with absolute paths to binaries and files, cd into the right directory, set needed variables in the crontab or a wrapper script, and always redirect output to a log with >> file 2>&1 so you can see errors. Also check the time zone.
Q4How do you prevent logs from filling the disk?
For text log files I add a logrotate rule per application: rotate daily or weekly, keep a fixed number of files, compress old ones, and use copytruncate or a postrotate signal so the app keeps writing to the new file. For the systemd journal I set SystemMaxUse in journald.conf so it stays under a size limit. Applications should avoid debug-level logging in production. For anything that matters long-term, logs should be shipped to a central system, such as Loki or a hosted service, so the server only keeps a short local window. And a disk usage alert catches whatever the rules miss.
Q5free shows almost no free memory. Is the server in trouble?
Usually not. Linux uses otherwise idle RAM as a page cache for files, which makes disk reads faster, and releases it immediately when applications need memory. That cache shows up as used, so the free column looks small on a healthy machine. The number to watch is available, which estimates how much memory can be given to new work without swapping. Real trouble looks like low available memory, steady swap activity, rising load with high I/O wait, or OOM killer messages in journalctl -k. Then I look for the process that grew, such as a leak or an untuned database, before simply buying more RAM.
Q6Should a server have swap?
On most small and medium VPS instances, yes, a modest amount, typically 1 to 2 GB or so. It acts as a safety buffer: a short memory spike makes the server slower instead of triggering the OOM killer, which might kill the database. It also lets the kernel move rarely used pages out of RAM. Swap is not extra memory, though; disk is far slower than RAM. If a server swaps constantly, it is undersized or an app is leaking, and the fix is more RAM or tuning. Some setups, like Kubernetes nodes or latency-critical databases, deliberately run without swap or with very low swappiness.
Senior-level questions
S1How do you balance automatic updates with stability on production servers?
I separate risk levels. Security updates from the distro's security pocket are applied automatically every day with unattended-upgrades, because a known vulnerability left open is a bigger risk than a well-tested security patch. Feature and point updates, third-party repositories such as databases or runtimes, and anything needing a restart of a critical service go through a planned window: staging first, a snapshot, then production. Packages that must move in step with the application, such as a database major version, are pinned or held. For kernel updates I either allow automatic reboots at a quiet hour, if the app tolerates a short restart, or track /var/run/reboot-required and reboot deliberately. Fleet-wide, I stagger updates across servers so a bad update never hits everything at once.
S2A database server is slow and iowait is high. How do you find out whether storage is the bottleneck?
I confirm it with numbers before tuning anything. vmstat shows blocked processes and wa; iostat -x shows per-device read and write latency and queue size. On network-attached cloud volumes, compare the observed IOPS and throughput with the plan's documented limits, because hitting a provider cap looks exactly like slow disk. pidstat -d or iotop confirms the database is the process doing I/O, and /proc/pressure/io shows how much time tasks are stalled. Then I check the cause on the database side: missing indexes causing full scans, a buffer pool too small for the working set, so reads go to disk, or a checkpoint or backup job saturating writes. Fixes range from query and memory tuning to local NVMe or a higher IOPS tier.
S3What would you keep on the server's disk, and what would you move off it?
I treat the server disk as replaceable. It holds the operating system, the application release, short-lived caches, a short window of local logs and the database files if the database is self-hosted. Anything whose loss would hurt goes elsewhere: user uploads to S3-compatible object storage, database backups to object storage at another provider or region, and logs shipped to a central system. That makes the disk small and predictable, so it rarely fills up, and makes rebuilding a server a matter of provisioning plus restoring data rather than rescuing a pet machine. The trade-offs are extra latency and egress for object storage, more moving parts, and the need to handle credentials for those services securely.
Glossary
- dpkg
- The low-level Debian package tool that installs .deb files and tracks their state. apt calls it.
- deb822 sources
- The structured key-value format for apt repository files (.sources), the default on Ubuntu since 24.04.
- Pinning
- apt preferences that give repositories or versions priorities, controlling which version gets installed.
- Phased updates
- Ubuntu's gradual rollout of non-security updates to a growing share of machines, to catch regressions early.
- Reserved blocks
- Space on ext4, 5% by default, that only root can use, keeping the system usable when a disk fills.
- Inode exhaustion
- Running out of file entries on a filesystem while free space remains, usually caused by millions of tiny files.
- LVM
- Logical Volume Manager: a layer that pools disks into volumes you can resize, span and snapshot.
- Page cache
- RAM the kernel uses to cache file data and buffer writes, released when applications need memory.
- I/O wait
- CPU time spent idle while processes wait for disk. High values point to storage, not CPU, as the bottleneck.
- Log shipping
- Sending logs from servers to a central store by an agent, so they are searchable and survive the server.
Keep learning
Have a product in mind?Let's build it.
Tell me about your idea, your product or your team. I usually reply within one working day.