Scaling and load balancing

01Vertical vs horizontal, and why stateless matters

Vertical scaling means a bigger machine: more CPU, RAM, faster disks. It is simple, needs no code changes and is often the right first step, but it has a ceiling, gets expensive, and one machine is still one point of failure. Horizontal scaling means more machines running identical copies of the app behind a load balancer. It scales further and survives the loss of a node, but only if any instance can serve any request. That requires a stateless app: sessions in Redis or signed tokens, uploads in object storage instead of local disk, cache in a shared store, scheduled jobs running once rather than on every node, and no in-memory state that matters. Make apps stateless early, even on one server, and horizontal scaling becomes straightforward.

• Sessions and cache: Redis, not process memory or local files.
• Files: S3-compatible object storage, served through a CDN.
• Cron: one scheduler or a distributed lock, so jobs do not run N times.

02Load balancers: L4 vs L7, algorithms, health checks

A load balancer spreads requests across instances. Layer 4 balancers work with TCP/UDP connections: very fast, protocol-agnostic, but blind to URLs and headers. Layer 7 balancers understand HTTP: they terminate TLS, route by host or path, add headers, retry failed requests, compress and rate limit. nginx, HAProxy, Envoy, Traefik and cloud application load balancers are common L7 choices. Algorithms include round robin (simple), least connections (better when requests vary in duration), weighted (mixed hardware) and hashing (same client to same server). Health checks remove broken instances: passive checks watch real traffic for errors, active checks poll a health endpoint. Sticky sessions pin a user to one instance; they hide statefulness and unbalance load, so prefer shared session storage and keep stickiness for cases like WebSocket fallbacks.

1upstream app {
2    least_conn;
3    server 10.0.0.11:3000 max_fails=3 fail_timeout=10s;
4    server 10.0.0.12:3000 max_fails=3 fail_timeout=10s;
5    keepalive 32;
6}
7 
8server {
9    listen 443 ssl;
10    server_name example.com;
11    # ssl_certificate / ssl_certificate_key lines go here
12    location / {
13        proxy_pass http://app;
14        proxy_http_version 1.1;
15        proxy_set_header Connection "";
16        proxy_set_header Host $host;
17        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
18        proxy_next_upstream error timeout http_502 http_503;
19    }
20}

03Caching layers: CDN, reverse proxy, application, database

The cheapest request is the one that never reaches your app. A CDN caches static assets and cacheable pages at edge locations near users, and also absorbs basic DDoS traffic. A reverse-proxy cache (nginx, Varnish) can serve repeated responses for a few seconds, which flattens spikes on hot pages. In the application, a Redis cache with the cache-aside pattern stores expensive query results with a TTL. The database has its own buffer cache, so good indexes keep hot data in memory. Each layer needs a clear invalidation story: version asset filenames, use short TTLs or purge on change for pages, and delete or update keys when data changes. Protect against stampedes when a popular key expires by using locks, request coalescing or slightly randomised TTLs.

04Scaling the database: pooling, replicas, partitioning, sharding

Databases are the usual bottleneck because they hold state and are hard to multiply. Work in order of cost. First, indexes and query fixes, found with slow-query logs and EXPLAIN. Second, connection pooling: PostgreSQL uses a process per connection, so dozens of app instances can exhaust it; PgBouncer in transaction mode lets thousands of client connections share a small pool. Third, read replicas for read-heavy traffic, remembering that replicas lag, so read-your-own-writes flows must hit the primary. Fourth, partitioning large tables (by date or tenant) inside one database to keep indexes small and drop old data cheaply. Last, sharding: splitting data across several databases by a shard key such as tenant ID. Sharding scales writes but makes cross-shard queries, transactions and rebalancing hard, so delay it until you have measured the need.

1; /etc/pgbouncer/pgbouncer.ini
2[databases]
3app = host=127.0.0.1 port=5432 dbname=app
4 
5[pgbouncer]
6listen_addr = 127.0.0.1
7listen_port = 6432
8auth_type = scram-sha-256
9auth_file = /etc/pgbouncer/userlist.txt
10pool_mode = transaction
11max_client_conn = 1000
12default_pool_size = 20

05Queues, back-pressure and capacity planning

Not everything must happen inside the request. Sending emails, generating reports, resizing images, syncing to an ERP: push them to a queue and let workers process them at a steady rate. Queues absorb spikes, isolate slow third parties and let you scale workers separately from web servers. Add back-pressure too: rate limits per client, timeouts on every outbound call, circuit breakers for failing dependencies, and graceful degradation (serve cached data, hide non-essential widgets) when overloaded. Capacity planning ties it together. Measure current throughput, latency percentiles (p95, p99) and resource usage; load test with tools like k6 to find the breaking point; identify which resource saturates first; then keep headroom (often 30 to 50 percent) for spikes and node failures. Repeat after major features, because bottlenecks move.

• Watch p95/p99 latency, not averages; averages hide the slow users.
• Load test a staging copy with realistic data volumes.
• Plan for losing one node at peak (N+1 capacity).

06Queueing theory you can use: Little's law and utilisation

Little's law links three numbers: requests in flight = arrival rate x average time in the system. At 200 requests per second and 250 ms per request, about 50 requests are in progress at any moment, which tells you how many workers, threads or database connections you need. The second lesson is non-linear: as a resource approaches full utilisation, waiting time explodes rather than growing steadily. A server at 50 percent busy has short queues; at 90 percent, small bursts create long queues and p99 latency jumps. That is why headroom matters and why averages mislead. It also explains why adding connections to an overloaded database makes it slower: more concurrent work means more contention, not more throughput. Size pools from Little's law with a margin, and cap concurrency so excess load waits in a queue you control.

• In flight = rate x latency; use it to size pools and workers.
• Plan to run key resources well below saturation at peak.
• Limit concurrency at the edge instead of letting it pile up in the database.

07Retry storms, timeouts budgets and load shedding

Retries help with brief glitches and hurt during overload. If each of three layers retries three times, one user request can become 27 calls to the bottom service exactly when it is struggling. Retry only at one layer, only idempotent operations, with exponential backoff and full jitter, and cap retries with a budget (for example, retries may add at most 10 percent extra load). Give every request a deadline and pass the remaining time downstream, so inner calls never outlive the user's patience. When overloaded, reject early with 429 or 503 and Retry-After instead of queueing forever; a fast failure is cheaper than a slow one. Prioritise traffic: shed analytics and recommendations before checkout. Circuit breakers stop calling a dependency that keeps failing and probe it periodically, giving it time to recover.

1# nginx: per-IP rate limit with a small burst, reject the rest fast
2limit_req_zone $binary_remote_addr zone=api:10m rate=20r/s;
3limit_req_status 429;
4 
5server {
6    location /api/ {
7        limit_req zone=api burst=40 nodelay;
8        proxy_connect_timeout 2s;
9        proxy_read_timeout 10s;
10        proxy_pass http://app;
11    }
12}

08Consistent hashing and cache scaling

When you spread keys across N cache or shard nodes with hash(key) mod N, adding one node remaps almost every key, so the cache empties and the database takes the full load at once. Consistent hashing places nodes and keys on a ring; each key belongs to the next node clockwise, so adding or removing a node moves only about 1/N of the keys. Virtual nodes (many points per server) smooth out the distribution. Redis Cluster uses a related idea: 16,384 hash slots assigned to nodes, moved in chunks during resharding, with hash tags like {tenant:42} to keep related keys in one slot. In load balancers, nginx's hash directive with the consistent parameter does the same for upstream selection. For hot keys that one node cannot serve, add a small in-process cache with a short TTL in front of the shared one.

09Tail latency and fan-out

When one request fans out to many backends, the slowest one decides the response time. If each backend has a 1 percent chance of a slow response, a request touching 100 of them is slow most of the time. Tail latency comes from garbage collection pauses, cold caches, noisy neighbours, lock contention and queueing. Defences: reduce fan-out by precomputing or denormalising; set per-call timeouts and return partial results when non-essential parts are late; use hedged requests, sending a second copy to another replica after the call exceeds the p95, only for idempotent reads; and keep instance sizes and runtime settings uniform so one weak node does not drag the whole pool. Always report p99 and p99.9 for key endpoints next to p50, and investigate the slowest traces, not just the averages.