Linux networking and SSH
IPs and ports, debugging connections, SSH keys done right, tunnels and a basic firewall.
In 30 seconds
A server talks to the world through network interfaces, IP addresses and ports. A few tools show you what is listening, test whether a URL or DNS record works, and let you log in securely with SSH keys. Add a simple firewall that only allows what you need, and you have covered most day-to-day networking on a Linux server.
Why I use it
- I manage every server over SSH with keys, and 'SSH keys only, no password login' is the first item of my security baseline.
- My firewall rule on servers is simple: ufw plus the provider's firewall, with only ports 22, 80 and 443 open.
- When a site is unreachable, tools like ss, curl and dig tell me in a minute whether it is DNS, the firewall, nginx or the app.
How it works
01Interfaces, IP addresses and ports
A network interface is a connection point: a physical or virtual card such as eth0 or ens3, plus lo, the loopback interface. Each interface has one or more IP addresses. 127.0.0.1 (and ::1 in IPv6) is localhost: it is only reachable from the machine itself, which makes it the right place for databases and app servers that sit behind nginx. Private ranges like 10.0.0.0/8 and 192.168.0.0/16 are used inside networks; a public IP is reachable from the internet. A port is a number from 1 to 65535 that identifies a service on an IP: 22 for SSH, 80 for HTTP, 443 for HTTPS, 5432 for PostgreSQL, 3306 for MySQL. An app listening on 0.0.0.0 accepts connections on every interface, including the public one.
ip -br addr # interfaces and IPs, brief
ip route # routing table and default gateway
hostname -I # all IPs of this host
ping -c 3 example.com # basic reachability (ICMP may be blocked)
traceroute example.com # path to a host (apt install traceroute)
02What is listening: ss, curl and DNS tools
ss has replaced netstat on modern distros. ss -tlnp shows TCP sockets that are listening, with numeric ports and the process that owns them; check that databases and internal services listen on 127.0.0.1, not 0.0.0.0. curl is the HTTP debugging tool: -I shows only headers, -v shows the whole conversation including TLS, and --resolve lets you test a new server before DNS points to it. wget is better for simply downloading files. dig queries DNS directly, which tells you whether a domain resolves to the server you expect. Asking a public resolver with @1.1.1.1 avoids your local cache.
sudo ss -tlnp # listening TCP ports + process
sudo ss -tnp state established # current connections
curl -I https://example.com # status and headers only
curl -v http://127.0.0.1:3000/health # talk to the app directly
curl --resolve example.com:443:203.0.113.10 https://example.com/
dig +short example.com A # IPv4 address
dig example.com MX # mail servers
dig @1.1.1.1 example.com # ask a specific resolver
dig +trace example.com # follow delegation from the root
03SSH keys: how they work and how to set them up
SSH gives you an encrypted shell on a remote machine. With key-based login you create a key pair on your own computer: a private key that never leaves it and a public key you copy to the server's ~/.ssh/authorized_keys. When you connect, the server checks that you hold the private key without it ever being sent. Use the ed25519 key type: it is short, fast and the modern default. Protect the private key with a passphrase and let ssh-agent remember it, so you type it once per session. The first time you connect, SSH shows the server's fingerprint and stores it in known_hosts; if that fingerprint changes later, SSH warns you, because it could mean someone is in the middle.
# On your laptop
ssh-keygen -t ed25519 -C "your_user@laptop"
ssh-copy-id [email protected] # installs the public key
ssh [email protected]
# If ssh-copy-id is not available, append the .pub file manually:
# cat ~/.ssh/id_ed25519.pub | ssh [email protected] "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
# Permissions on the server must be strict:
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
04~/.ssh/config and disabling password login
A ~/.ssh/config file on your laptop gives each server a short name with its user, address, key and options, so ssh web1 replaces a long command. On the server, once your key works, turn off password logins and direct root logins; that removes the brute-force attacks that hit every public SSH port within minutes. On current Ubuntu, put your settings in a file under /etc/ssh/sshd_config.d/, which is included before the main file (the first value set wins). Always validate with sshd -t and keep your current session open while you test a new one, so a mistake cannot lock you out. Some cloud images add their own file in that directory that re-enables passwords, so check it.
# ~/.ssh/config on your laptop
Host web1
HostName 203.0.113.10
User your_user
IdentityFile ~/.ssh/id_ed25519
# /etc/ssh/sshd_config.d/10-hardening.conf on the server
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
# Then: sudo sshd -t && sudo systemctl reload ssh (sshd on RHEL)
05Copying files: scp and rsync
scp copies files over SSH with a cp-like syntax and is fine for a single file. rsync is better for anything bigger: it only transfers the differences, can resume, preserves permissions and times with -a, and can mirror a directory with --delete. The trailing slash matters: rsync src/ dest/ copies the contents of src, while rsync src dest/ creates dest/src. Always run a destructive rsync with --dry-run (-n) first to see what would change. Both tools use your SSH config, so host aliases work.
scp ./dump.sql web1:/tmp/ # one file up
scp web1:/var/log/nginx/error.log . # one file down
rsync -avz --progress ./dist/ web1:/srv/myapp/dist/
rsync -avzn --delete ./dist/ web1:/srv/myapp/dist/ # dry run first!
rsync -avz web1:/var/www/app/storage/app/ ./storage-backup/
06Port forwarding and tunnels
SSH can carry other traffic through its encrypted connection. Local forwarding (-L) is the most useful: it opens a port on your laptop that leads to a port on the server, so you can reach a database or admin panel that only listens on the server's localhost, without ever opening it to the internet. Remote forwarding (-R) does the reverse and exposes a port of your laptop on the server, for example to demo a local app. -D creates a SOCKS proxy. A jump host (ProxyJump or -J) lets you reach private servers through one public bastion. Add -N when you only want the tunnel and no shell.
# Reach the server's local PostgreSQL at localhost:5433 on your laptop
ssh -N -L 5433:127.0.0.1:5432 web1
# Expose your local port 3000 as port 8080 on the server's localhost
ssh -N -R 8080:127.0.0.1:3000 web1
# Reach a private server through a bastion
ssh -J [email protected] [email protected]
# Or in ~/.ssh/config: ProxyJump bastion
07Firewall basics with ufw
A firewall decides which incoming and outgoing connections are allowed. On Ubuntu, ufw (uncomplicated firewall) is a friendly front end to the kernel's netfilter rules; RHEL uses firewalld. The safe default is to deny all incoming traffic, allow outgoing, and open only what you serve: SSH, HTTP and HTTPS. Allow SSH before you enable the firewall, or you will cut off your own session. Use the provider's cloud firewall as a second layer: it blocks traffic before it reaches the server. One important gotcha: Docker publishes container ports by writing its own iptables rules, which bypass ufw, so bind containers to 127.0.0.1 unless they really must be public.
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH # allow 22 BEFORE enabling
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw limit OpenSSH # rate-limit repeated SSH attempts
sudo ufw enable
sudo ufw status verbose
sudo ufw allow from 10.0.0.0/16 to any port 5432 # DB from private net only
Use it when
- Setting up any new server: keys, no password login, firewall, before deploying anything.
- Debugging 'site is down' reports: DNS, port, TLS or app, layer by layer.
- Reaching databases and internal tools securely through SSH tunnels instead of exposing them.
- Deploying or backing up files with rsync over SSH.
Skip it when
- Do not expose database ports to the internet just to connect a GUI client; use an SSH tunnel.
- Do not rely on changing the SSH port as your security; it only reduces log noise. Keys and the firewall do the work.
- For teams with many servers, raw shared keys get messy; consider SSH certificates or an access tool instead.
Alternatives
- firewalld / nftables
firewalld is the default on RHEL; nftables is the modern kernel rule system that both ufw and firewalld can use.
- Cloud provider firewalls
Filter traffic before it reaches the VM. Use them together with a host firewall.
- WireGuard / Tailscale
Private VPN mesh so admin ports are never public at all.
- Session managers
Tools like AWS Systems Manager or Teleport give audited shell access without open SSH ports.
Best practices
- Use ed25519 keys with a passphrase and ssh-agent; one key per device so you can revoke one.
- Disable PasswordAuthentication and PermitRootLogin, and verify with sshd -t before reloading.
- Keep a second SSH session open while changing SSH or firewall settings.
- Default-deny incoming; open only 22, 80 and 443, plus the provider firewall as a second layer.
- Bind databases, Redis and app servers to 127.0.0.1 or a private network, never 0.0.0.0 on a public host.
- Add fail2ban to ban IPs that keep failing SSH or web logins.
- Run rsync with --dry-run before any --delete.
Common mistakes
- Enabling ufw before allowing SSH and locking yourself out of the server.
- Disabling password login before checking that the key actually works.
- Publishing Docker ports (-p 5432:5432) and assuming ufw protects them.
- Loose permissions on ~/.ssh or authorized_keys, so SSH silently ignores the key.
- Ignoring a 'host key changed' warning and deleting the known_hosts entry without checking why.
Deep dive
08TCP connection states and what they tell you
Every TCP connection moves through states, and counting them is a fast health check. A server socket sits in LISTEN; the kernel completes the three-way handshake and queues new connections in the accept backlog until the app accepts them. ESTABLISHED connections carry data. The side that closes first ends in TIME_WAIT for about a minute so delayed packets cannot corrupt a new connection; thousands of TIME_WAIT sockets on a busy proxy are normal. CLOSE_WAIT is different: the peer has closed, but your application has not closed its end. A growing number of CLOSE_WAIT sockets means a bug, typically a leaked connection in a pool, and eventually the process runs out of file descriptors. When the backlog overflows, new clients see timeouts even though the process looks healthy; ss shows the queue sizes on listening sockets.
ss -s # summary of sockets by state
ss -tan state close-wait | wc -l # leaked connections?
ss -tan state time-wait | wc -l
ss -ltn # Recv-Q/Send-Q = backlog use/size
sysctl net.core.somaxconn net.ipv4.ip_local_port_range
nstat -az | grep -i -E 'ListenOverflows|ListenDrops'
09Inside an SSH connection, and SSH certificates
An SSH session starts with key exchange: client and server agree on algorithms and derive session keys, and the server proves its identity with its host key, which the client checks against known_hosts. Since OpenSSH 10.0 (April 2025) the default key exchange is mlkem768x25519-sha256, a hybrid of classic X25519 and post-quantum ML-KEM, and DSA keys are gone entirely. Only then does user authentication happen, by key, certificate or other methods. For teams, plain authorized_keys files scale badly: every server has a list to maintain and revoking a person means editing all of them. SSH certificates solve this. An internal certificate authority signs a user's public key with an identity, allowed login names and an expiry, and servers trust the CA through one line, TrustedUserCAKeys. Short-lived certificates expire on their own, so offboarding is mostly automatic.
ssh -Q kex # key exchange algorithms supported
ssh -v web1 2>&1 | grep -i 'kex: algorithm'
# Certificate authority (keep the CA private key offline or in a vault)
ssh-keygen -t ed25519 -f user_ca -C "user CA"
ssh-keygen -s user_ca -I [email protected] -n your_user -V +8h ~/.ssh/id_ed25519.pub
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub # inspect the certificate
# On each server, in sshd_config.d:
# TrustedUserCAKeys /etc/ssh/user_ca.pub
10Netfilter: where ufw, Docker and nftables meet
All Linux firewalls are front ends to netfilter, the packet filtering framework in the kernel. Packets pass through hooks: PREROUTING when they arrive, then INPUT if they are for this host or FORWARD if they are routed elsewhere, OUTPUT for locally generated packets and POSTROUTING on the way out. ufw only manages INPUT-style rules for the host itself. Docker publishes container ports with DNAT in PREROUTING, which turns the traffic into forwarded traffic that never meets ufw's INPUT rules; that is the mechanism behind the well-known bypass, and why the DOCKER-USER chain exists. Connection tracking (conntrack) remembers flows, so rules can allow established replies cheaply; on very busy hosts the conntrack table can fill and drop new connections. On current distros the iptables command usually writes nftables rules underneath, and nft list ruleset shows everything in one place.
sudo nft list ruleset | less # every rule, all tools
sudo iptables -L DOCKER-USER -n -v # Docker's filter hook
# Allow only one network to reach published containers
sudo iptables -I DOCKER-USER -i eth0 ! -s 198.51.100.0/24 -j DROP
sudo conntrack -C # tracked connections (conntrack pkg)
sysctl net.netfilter.nf_conntrack_max
11How the host resolves names
dig talks to DNS servers directly, but applications use the system resolver, and the two can disagree. The C library follows /etc/nsswitch.conf, which usually says: check /etc/hosts first, then DNS. On Ubuntu, DNS goes to systemd-resolved, a local caching stub listening on 127.0.0.53, which forwards to the upstream servers configured by netplan or DHCP. So an app can see an old address from the cache or an /etc/hosts override while dig shows the new one. getent hosts uses exactly the same path as applications, which makes it the right test. resolvectl shows the upstream servers per interface and can flush the cache. Languages add their own caching too: the JVM caches lookups, and connection pools keep using an old IP until they reconnect, which matters during migrations.
getent hosts example.com # what applications will see
resolvectl status # upstream DNS per interface (Ubuntu)
resolvectl query example.com # resolve through systemd-resolved
sudo resolvectl flush-caches
grep hosts /etc/nsswitch.conf # lookup order
cat /etc/resolv.conf # usually points to 127.0.0.53
Real-world scenarios
Moving SSH to another port on Ubuntu 24.04
Problem
To cut log noise you change Port in an sshd_config.d file and restart ssh, but the server keeps listening on 22. On another attempt you nearly lock yourself out.
How to solve it
Since Ubuntu 22.10, SSH is socket-activated: systemd's ssh.socket owns the listening port, and the port is generated from sshd_config at daemon-reload time. Open the new port in ufw and the provider firewall first, keeping 22 open. Add Port 2222 (alongside Port 22 for now) in /etc/ssh/sshd_config.d/, run sudo sshd -t, then sudo systemctl daemon-reload and sudo systemctl restart ssh.socket. Check with ss -tlnp and log in on the new port from a second terminal. Only then remove port 22 from the config and firewall. Remember a changed port only reduces noise; keys, no passwords and fail2ban are the actual protection.
Connection refused versus connection timed out
Problem
A new API on port 8080 is unreachable from the office. Some colleagues report 'connection refused', others see the request hang until it times out.
How to solve it
The two errors point to different layers. Refused means a host actively answered with a reset: the packet arrived, but nothing listens on that address and port, often because the app is bound to 127.0.0.1 or crashed; check ss -tlnp on the server. A timeout means packets are dropped silently, which is a firewall (ufw, the provider firewall, an office network) or routing problem. Test from the server itself with curl localhost:8080, then from another server in the same region, then from outside. Run tcpdump -ni eth0 port 8080 while testing to see whether packets arrive at all. Usually the right fix is not opening 8080, but proxying the API through nginx on 443.
The app stops accepting connections after a few hours
Problem
A Node service works after each restart but becomes unresponsive within hours. CPU is low, and the logs show 'EMFILE: too many open files'.
How to solve it
Count sockets by state: ss -tanp | grep node | awk '{print $1}' | sort | uniq -c. Thousands in CLOSE_WAIT mean the app is not closing connections whose peer already closed, typically an HTTP client or database pool leaking connections, or responses that are never consumed. Each socket is a file descriptor, so the process hits its limit, shown in /proc/PID/limits. Raise LimitNOFILE in the unit as a buffer, but fix the code: reuse a single HTTP agent with keep-alive and sane timeouts, always release pooled connections in finally blocks, and set server timeouts. Add a metric for open file descriptors so the next leak is caught early.
Cheat sheet
- Socket summary by state
ss -s- Which process owns port 443
sudo ss -ltnp 'sport = :443'- Watch packets on a port
sudo tcpdump -ni any port 443 -c 20- Test a TCP port without curl
nc -zv 203.0.113.10 5432- Check the certificate a server presents
openssl s_client -connect example.com:443 -servername example.com </dev/null- What applications resolve a name to
getent hosts example.com- Show the full firewall ruleset
sudo nft list ruleset- Reuse one SSH connection for many sessions
ControlMaster auto / ControlPath ~/.ssh/cm-%r@%h:%p / ControlPersist 10m- Apply an SSH port change on Ubuntu 24.04
sudo sshd -t && sudo systemctl daemon-reload && sudo systemctl restart ssh.socket- Ban status of fail2ban's SSH jail
sudo fail2ban-client status sshd
In my real projects
- I manage my servers over SSH with key-based login, and my baseline is SSH keys only with password login disabled.
- My firewall standard is ufw plus the provider firewall, with only 22, 80 and 443 open, and databases bound to localhost or a private network.
- fail2ban and ufw are present on the servers I run, and Cloudflare sits in front of some sites for DNS.
Questions & answers
Q1How does SSH key authentication work, and why is it better than passwords?
You generate a key pair. The public key goes into the server's authorized_keys; the private key stays on your machine. When you connect, the server sends a challenge that only the holder of the private key can sign, and verifies the signature with the public key. The private key never crosses the network. Passwords can be guessed, reused or phished, and every public SSH port receives constant brute-force attempts. A key has far more entropy than any password, and when protected with a passphrase it is also useless to someone who steals the file. You can also revoke one device's key without affecting others.
Q2A website is not loading. How do you troubleshoot it from the server side?
I go layer by layer. DNS first: dig +short example.com should return the server's IP. Then reachability: curl -I from outside, and check the provider and ufw firewalls allow 80 and 443. On the server, ss -tlnp shows whether nginx is listening on those ports, and systemctl status nginx and nginx -t show whether it is running with a valid config. Next I bypass nginx and curl the app directly on its local port. If that fails, the app's journal or log tells me why. If it works, the problem is the proxy config or TLS, which curl -v shows clearly.
Q3What does ss -tlnp show, and why do you care whether a service listens on 0.0.0.0?
ss lists sockets: -t for TCP, -l for listening ones, -n for numeric ports instead of names and -p for the owning process. It answers 'what is open on this machine and who opened it'. The address matters: 127.0.0.1 means only local processes can connect, 0.0.0.0 (or [::]) means every interface, including the public one. A database or Redis listening on 0.0.0.0 on an internet-facing host is one firewall mistake away from being exposed, and exposed databases are found and attacked by scanners within hours. So I bind internal services to localhost or a private network interface.
Q4How would you connect to a production database that is not exposed to the internet?
Through an SSH tunnel. ssh -N -L 5433:127.0.0.1:5432 your_user@server opens port 5433 on my laptop, and anything connecting to it travels inside the encrypted SSH session and comes out on the server as a local connection to PostgreSQL. My database client points at localhost:5433. The database keeps listening only on localhost, so nothing is opened in the firewall. If the database runs on a separate private host, I tunnel through the app server or a bastion with ProxyJump. For a team, a VPN like WireGuard or a managed access tool can replace ad-hoc tunnels.
Q5Does ufw protect ports published by Docker?
Not by default, which surprises many people. When you publish a port with -p 8080:80, Docker inserts its own iptables rules that forward traffic to the container before ufw's rules are checked, so the port is reachable from the internet even if ufw denies it. The simplest fix is to publish only on localhost, for example -p 127.0.0.1:8080:80, and put nginx in front. Docker also supports filtering in the DOCKER-USER chain, and a cloud provider firewall in front of the server blocks the traffic regardless. I always verify from outside with a port scan or curl.
Q6What is the difference between scp and rsync?
Both copy files over SSH. scp is a simple copy, like cp across machines: it always transfers the whole file and has few options. rsync compares source and destination and sends only what changed, so repeated syncs of large directories are fast. It can resume interrupted transfers, preserve permissions, owners and timestamps with -a, compress with -z, exclude patterns, and mirror a directory exactly with --delete. That makes it the right tool for deployments of static builds and for backups. I always run it with --dry-run first when --delete is involved, and I pay attention to the trailing slash on the source path.
Senior-level questions
S1How would you give a team of engineers SSH access to dozens of servers?
Copying personal public keys to every server works for two people and five servers; beyond that, onboarding and especially offboarding become error-prone. The better model is central identity with short-lived credentials. SSH certificates signed by an internal CA, issued for a few hours after single sign-on, mean servers trust one CA key and nothing needs cleaning up when someone leaves. Tools like Teleport, cloud session managers or HashiCorp Vault's SSH engine automate issuing and add session recording. Put servers on a private network reachable through a bastion or a VPN mesh like WireGuard or Tailscale, so SSH is not exposed publicly at all. The trade-offs are operating the CA or tool itself and having a break-glass path if it is down.
S2Why is SSH agent forwarding risky, and what do you use instead?
With agent forwarding (-A), the remote server gets a socket to your local ssh-agent. Your private key never leaves your laptop, but anyone with root on that server, or access to your user there, can use the socket to authenticate as you to any other host while you are connected. On a shared or compromised jump host that is a real lateral-movement risk. For reaching internal servers, ProxyJump (-J) is the safer replacement: the connection to the target is tunneled through the bastion, but authentication happens end to end from your laptop, so the bastion never touches your keys. For pulling from Git on a server, use a read-only deploy key specific to that repository rather than forwarding your personal agent.
S3Host firewall, cloud firewall, or both? How do you think about network defense on a VPS?
Both, because they fail differently. The provider's cloud firewall filters traffic before it reaches the machine, survives a misconfigured or flushed host ruleset, and is not bypassed by Docker's iptables rules. The host firewall works regardless of provider, can express finer rules such as per-interface or per-source limits, and keeps protecting the host if the cloud rule set is edited by mistake. Neither replaces binding internal services to localhost or a private interface, which removes exposure entirely instead of filtering it. On top come rate limits and fail2ban for login endpoints, and a CDN or proxy in front of web traffic for DDoS absorption, with the origin accepting web traffic only from the proxy's ranges. The cost is keeping two rule sets consistent, ideally in code.
Glossary
- Three-way handshake
- SYN, SYN-ACK, ACK: the exchange that opens a TCP connection before any data is sent.
- Accept backlog
- The kernel queue of completed connections waiting for the application to accept them. Overflow causes client timeouts.
- TIME_WAIT
- State kept briefly by the side that closed first, so late packets cannot confuse a new connection. Normal in bulk.
- CLOSE_WAIT
- The peer closed but the local application has not. Many of these indicate a connection leak in the app.
- Host key
- A server's own SSH key pair. Clients record its fingerprint to detect impersonation on later connections.
- SSH certificate
- A user or host key signed by a trusted CA, with names and an expiry, replacing per-server key lists.
- Netfilter
- The Linux kernel framework that filters, translates and tracks packets. iptables, nftables, ufw and firewalld configure it.
- DNAT
- Destination NAT: rewriting a packet's destination address or port, used by Docker to forward published ports to containers.
- conntrack
- Netfilter's table of active flows, letting rules allow reply traffic and NAT work. It has a size limit.
- Stub resolver
- A local DNS forwarder, like systemd-resolved on 127.0.0.53, that caches answers and forwards queries upstream.
Keep learning
Have a product in mind?Let's build it.
Tell me about your idea, your product or your team. I usually reply within one working day.