Linux basics: the shell, files and pipes

01What Linux and a distribution are

Strictly speaking, Linux is only the kernel: the core program that talks to the hardware, schedules processes and manages memory. A distribution (distro) bundles that kernel with a package manager, system tools, a default configuration and a release schedule. On servers you will mostly meet two families. The Debian family (Debian, Ubuntu) uses .deb packages and apt. The Red Hat family (RHEL, Rocky Linux, AlmaLinux, Fedora) uses .rpm packages and dnf. For servers, pick a long-term support (LTS) release: Ubuntu LTS versions come out every two years and get five years of standard security updates, Debian stable gets around five years with its LTS extension, and RHEL-compatible releases get about ten. Almost everything in this lesson works the same on every distro; the main differences are package names, the package manager and a few config file locations.

• Kernel: the core of the OS. Distro: kernel plus tools, packages and defaults.
• Debian / Ubuntu: apt and .deb. RHEL / Rocky / Alma: dnf and .rpm.
• Shell: the program that reads your commands. Bash is the default on most servers.

02The filesystem hierarchy: where things live

Linux has one tree of directories starting at / (the root). There are no drive letters: extra disks are attached (mounted) somewhere inside that tree. The layout follows a standard called the Filesystem Hierarchy Standard, so once you learn it you can find your way around any server. A useful rule: configuration lives in /etc, things that change while the system runs (logs, caches, databases, websites) live in /var, programs live in /usr, and people live in /home. Everything is a file, including devices (/dev) and live kernel information (/proc, /sys), which is why so many tools are just about reading and writing text.

• /etc: system and service configuration (nginx, ssh, cron, users).
• /var: changing data. /var/log for logs, /var/www for websites by convention, /var/lib for databases and app state.
• /usr: installed programs and libraries. /usr/local for things you install by hand.
• /home/your_user: personal files. /root is the root user's home.
• /tmp: temporary files, often cleared on reboot. /opt: optional, self-contained apps.
• /dev, /proc, /sys: devices and live kernel and process information.

03Navigating and managing files

A handful of commands cover most daily work. pwd prints where you are, cd moves you, ls lists. Paths starting with / are absolute; anything else is relative to the current directory. A single dot means here, two dots mean the parent and a tilde means your home. Be careful with rm: there is no recycle bin, and rm -r deletes a whole tree without asking. A good habit is to run ls on the exact same path first, and to never paste an rm command with a variable you have not checked. find is the Swiss army knife for locating files by name, size, age or type.

1pwd                          # where am I?
2ls -lah /var/log             # long list, all files, human sizes
3cd /etc/nginx && ls          # move, then list
4cd ~ ; cd -                  # go home, then back to the previous dir
5mkdir -p ~/backups/2026      # create a path, including parents
6cp -a site/ site-backup/     # copy recursively, keep owners and times
7mv old.conf old.conf.bak     # move or rename
8rm -i notes.txt              # delete, asking first
9rm -r build/                 # delete a directory tree (no undo!)
10find /var/www -name "*.log" -size +100M     # big log files
11find /tmp -type f -mtime +7                 # files older than 7 days

04Reading files, pipes and redirection

cat prints a whole file, less lets you scroll and search one (press / to search, q to quit), head and tail show the start or end, and tail -f keeps following a file as new lines arrive, which is perfect for watching a log while you reproduce a bug. The real power comes from pipes. Every program has standard input, standard output and standard error. The pipe character | sends one program's output into the next one's input, so small tools combine into answers. Redirection sends output to files: > overwrites, >> appends, and 2> captures errors separately. 2>&1 merges errors into normal output.

1less /var/log/syslog                 # scroll and search
2tail -n 100 -f /var/log/nginx/error.log   # follow live
3# Top 10 client IPs in an access log
4awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
5# Save output, append to a file, capture errors
6df -h > disk.txt
7date >> deploy.log
8./deploy.sh > deploy.out 2>&1
9# Count 500 errors
10grep -c '" 500 ' /var/log/nginx/access.log

05grep, awk and sed in five minutes

These three tools handle most text problems. grep finds lines that match a pattern: -i ignores case, -r searches directories, -n shows line numbers, -v inverts the match, and -E enables extended regular expressions. awk splits each line into fields ($1, $2…) and is ideal for pulling columns out of logs. sed edits streams of text, most often for search and replace. Always run sed without -i first to preview the result, and when you do edit in place, keep a backup with -i.bak. For anything longer than one line of awk or sed, a small script in a real language is usually clearer.

1grep -rn "DB_HOST" /var/www/app/config   # where is it set?
2grep -i "error" app.log | grep -v "deprecated"
3grep -E "timeout|refused" app.log
4awk -F: '{print $1}' /etc/passwd             # first column, ":" separator
5awk '$9 >= 500 {print $7}' access.log         # paths that returned 5xx
6sed 's/http:/https:/g' links.txt              # preview a replacement
7sed -i.bak 's/^#Port 22/Port 22/' my.conf     # edit in place, keep .bak

06Editors, help and environment variables

On a server you edit files in the terminal. nano is the friendliest: the shortcuts are listed at the bottom (Ctrl+O saves, Ctrl+X exits). vim is everywhere and worth knowing at a survival level: press i to insert text, Esc to stop, then :wq to save and quit or :q! to quit without saving. When you forget a flag, use command --help for a short summary or man command for the full manual. Environment variables are named values that programs inherit from the shell, such as PATH (where commands are searched for) and HOME. Set one for the current session with export; make it permanent in ~/.bashrc. Apps should read secrets from environment variables or an env file, never from code.

1nano /etc/hosts        # simple editor
2vim  /etc/hosts        # i = insert, Esc, :wq = save & quit
3ls --help | less       # quick help
4man rsync              # full manual (q to quit)
5echo $PATH             # where commands are found
6export APP_ENV=production
7printenv | grep APP_   # show matching variables
8which node             # which binary runs?

07Your first shell script

A shell script is a text file of commands that runs top to bottom. The first line (the shebang) says which interpreter to use. Start every bash script with set -euo pipefail: -e stops on the first failed command, -u treats unset variables as errors, and pipefail makes a pipeline fail if any part fails. Without these, a script can keep going after a failure and do damage, for example deleting files in the wrong directory because a variable was empty. Quote your variables, make the file executable with chmod +x, and test it with shellcheck if you can install it.

1#!/usr/bin/env bash
2set -euo pipefail
3 
4# Archive a directory with today's date
5SRC="/var/www/app/storage"
6DEST="$HOME/backups"
7STAMP="$(date +%F)"
8 
9mkdir -p "$DEST"
10tar -czf "$DEST/storage-$STAMP.tar.gz" -C "$SRC" .
11echo "Saved $DEST/storage-$STAMP.tar.gz"
12 
13# Delete archives older than 14 days
14find "$DEST" -name "storage-*.tar.gz" -mtime +14 -delete

08What the shell really does when you press Enter

Before any program runs, bash rewrites your line in a fixed order: brace expansion, tilde expansion, then parameter, command and arithmetic substitution, then word splitting, then filename globbing, and finally quote removal. Only then does it look the command up: aliases, functions and builtins (cd, export, read) first, then each directory in PATH, with results cached in a hash table. For an external program the shell forks a copy of itself and the child calls exec to replace itself with the program; the parent waits and stores the exit status in $?. This explains classic surprises: cd must be a builtin because a child cannot change its parent's directory, an unquoted variable containing spaces becomes several arguments, and a glob that matches nothing is passed through literally. type and command -v tell you exactly what will run.

1type cd ls grep           # builtin, alias or file?
2command -v python3        # path that will be executed
3hash -r                   # forget cached paths after moving a binary
4false; echo $?            # 1: last exit status
5f="my file.txt"
6ls $f                     # WRONG: two arguments, "my" and "file.txt"
7ls "$f"                   # right: one argument
8set -x                    # print each command after expansion (debug)

09Inodes, hard links and symlinks

A file name is not the file. The data and metadata (owner, mode, timestamps, size, block pointers) live in an inode, identified by a number; a directory is just a table mapping names to inode numbers. A hard link is a second name for the same inode, so both names are equal and the data survives until the last link is removed and no process holds it open. Hard links cannot cross filesystems or point to directories. A symbolic link is a tiny file containing a path, which can point anywhere, even to nothing (a dangling link). This model explains why mv within one filesystem is instant (only a directory entry changes) while mv across filesystems copies data, why deleting an open log frees no space, and why deploy tools switch releases atomically by replacing a 'current' symlink.

1ls -li /etc/hostname          # inode number in the first column
2stat /etc/hostname            # full metadata, link count, times
3ln data.csv data-copy.csv     # hard link: same inode
4ln -s /srv/app/releases/42 /srv/app/current.new
5mv -T /srv/app/current.new /srv/app/current   # atomic symlink swap
6readlink -f /srv/app/current  # resolve to the real path
7find /srv -xtype l            # dangling symlinks

10Scripts that survive real data

Production scripts meet file names with spaces, newlines and leading dashes, interrupted runs and partial failures. Three habits handle most of it. First, pass file lists NUL-separated: find -print0 with xargs -0 or a while read -d '' loop, never a for loop over ls output. Second, create temporary files with mktemp and remove them in a trap, so cleanup happens even when the script fails or is interrupted. Third, know where set -e does not fire: inside if conditions, on the left of && or ||, and in command substitutions assigned with local, so check critical steps explicitly. Use -- to end options before user-supplied paths, so a file called -rf is treated as a name. For locking, flock prevents two copies of the same job from running at once.

1#!/usr/bin/env bash
2set -euo pipefail
3tmp="$(mktemp -d)"
4trap 'rm -rf -- "$tmp"' EXIT          # always cleaned up
5exec 9>/var/lock/report.lock
6flock -n 9 || { echo "already running" >&2; exit 1; }
7 
8find /var/www/uploads -type f -name '*.jpg' -mtime -1 -print0 |
9  while IFS= read -r -d '' file; do
10    cp -- "$file" "$tmp/"
11  done
12tar -czf /var/backups/uploads-today.tar.gz -C "$tmp" .

11Fast text processing on big files

On a multi-gigabyte log the choice of tool and flags matters. Locale is the hidden cost: with a UTF-8 locale, sort and grep do expensive character-aware comparisons; prefixing LC_ALL=C makes them compare bytes and often several times faster, which is fine for logs and IPs. grep -F treats the pattern as a fixed string and skips the regex engine. ripgrep (rg) searches in parallel and is usually fastest for code trees. Read compressed rotated logs in place with zgrep, zcat or zless instead of unpacking them. sort can use more memory with -S and parallel threads with --parallel, and writes temp files to /tmp unless you set -T, which matters when /tmp is small. Finally, put the most selective filter first in a pipeline so later stages process fewer lines.

1LC_ALL=C grep -F ' 502 ' access.log | wc -l
2zgrep -h 'POST /login' /var/log/nginx/access.log.*.gz | wc -l
3LC_ALL=C sort -S 1G --parallel=4 -T /var/tmp big.txt | uniq -c
4rg -n --hidden -g '!node_modules' 'API_KEY' /srv/app
5# Requests per minute around an incident
6awk '{print substr($4, 2, 17)}' access.log | uniq -c | sort -rn | head