Production infrastructure: the best setup for each stage

01Principles before diagrams

Good infrastructure is boring: predictable deploys, clear ownership, and failures that are noticed and recoverable. Before picking tools, answer four questions. How much downtime can the business tolerate (availability target)? How much data can it lose (RPO) and how fast must it recover (RTO)? Who will operate it at 3 a.m.? What is the monthly budget? Then grow in steps. Every step should remove a real bottleneck or risk you can name: the database fighting the app for memory, uploads stuck on one disk, a single server being a single point of failure, or a team deploying dozens of services. Moving to the next stage too early adds cost and complexity without benefit; moving too late means outages. Whatever the stage, keep infrastructure described in code or at least in written runbooks, so it can be rebuilt.

02Stage 1: a single VPS done right

For an MVP, a small business site or an internal tool, one well-configured server (2 to 8 GB RAM) is often the best choice: cheap, fast, easy to understand. nginx terminates HTTPS and routes to the apps; PHP-FPM and Node run under their own users; the database and Redis listen on localhost; workers run under supervisor or systemd. Docker Compose is a good option to package it all reproducibly. What makes it 'done right' is everything around the apps: the hardening baseline, Cloudflare in front, nightly database dumps copied off the server, provider snapshots, uptime and disk alerts, and a scripted deploy with release folders. Its weakness is that it is a single point of failure, so recovery speed matters: you should be able to rebuild it from scripts and backups in hours, and you should have practised that once.

• Cloudflare (DNS, WAF, CDN) → nginx → app processes → local DB/Redis.
• Backups off-site, snapshots on, restores tested.
• Typical cost: tens of dollars per month.

03Stage 2: app server + managed database + object storage + CDN

When the data becomes valuable or the database needs more memory, move state off the app server. A managed database brings automated backups, point-in-time recovery and an optional standby. User uploads go to S3-compatible object storage instead of the local disk, served through a CDN. Sessions and cache go to Redis, managed or on a private server. The app server now holds no unique data, which is the key change: you can rebuild or replace it at any time, resize it with little risk, and later add a second one. Configuration and secrets come from environment variables, and deploys build the same artifact for staging and production. This stage is the sweet spot for many profitable products and can carry a surprising amount of traffic with good caching and indexes.

• App server is stateless: code + config only.
• Database: managed, private network access only, PITR on.
• Files: object storage + CDN; no uploads on server disk.

04Stage 3: load balancer, several app servers, replicas and queues

When one app server is no longer enough, or the business needs to survive a server failure, put a load balancer in front of two or more identical app servers, ideally in different availability zones. The load balancer health-checks them and stops sending traffic to a broken one. Deploys roll through servers one at a time. The database gets read replicas for heavy read traffic and reports, and a standby for failover. Slow work (emails, PDFs, image processing, webhooks, imports) moves to background queues with dedicated worker servers, so web requests stay fast and spikes are absorbed. Watch for hidden state: file uploads, sessions, scheduled jobs (run the scheduler on one node only or use a lock), and in-memory caches must all be shared or coordinated. Replication lag also means a user may not instantly read what they just wrote from a replica.

05Stage 4: containers and Kubernetes

Containers package each service with its dependencies, so the same image runs on a laptop, in CI and in production. Docker Compose on one or a few servers covers a lot of ground. Kubernetes (or a managed alternative such as ECS, Cloud Run or a managed Kubernetes service) adds scheduling across many machines, self-healing, rolling updates, autoscaling and service discovery. It pays off when you run many services, several teams deploy independently, or traffic varies strongly. The price is real: a platform to learn and operate, more YAML, more ways to misconfigure networking and security, and usually a higher bill. A managed control plane removes some pain but not the complexity of the workloads. Many successful products never need it; adopt it when the coordination cost of not having it is clearly higher.

06Environments, secrets, DR, monitoring and cost

Every stage needs at least a staging environment that mirrors production in versions and configuration, with its own database and credentials, so changes are tested before customers see them. Secrets are injected per environment from env files or a secrets manager, never shared between staging and production, never in Git. Disaster recovery means written, rehearsed steps: where the backups are, how to rebuild a server, how to fail over the database, who decides. Monitoring has three parts: metrics (CPU, memory, disk, latency, error rates, queue length, for example with Prometheus and Grafana), logs collected centrally (Loki or a hosted service) and alerts that wake someone only for real user impact, plus external uptime checks. Review cost monthly: unused servers, oversized instances, old snapshots and egress fees add up quietly.

• dev → staging → production, same build artifact promoted forward.
• Alert on symptoms (errors, latency, downtime), investigate with metrics and logs.
• Tag resources by project and environment to see where money goes.

07Migrating or merging servers without losing data

Moving sites to a new server, or merging several servers into fewer, is mostly careful bookkeeping. Inventory first: every domain, app, database, cron job, worker, certificate, mailbox, environment file and upload folder on the old server. Build the new server from your baseline, then copy code, files and databases, and verify: run the apps against a hosts-file override, compare row counts and file counts, check logs. Lower the DNS TTL a day before. At cutover, briefly put the old app in maintenance mode, run a final incremental file sync and a fresh database dump, import it, then switch DNS (instant with Cloudflare proxy). Keep the old server running, untouched, until traffic, data and backups on the new one are verified for days; only then decommission it, after a final backup.

1# 1. Initial copy of code and uploads (preserves owners, permissions, links)
2rsync -aHAX --info=progress2 [email protected]:/var/www/shop/ /var/www/shop/
3 
4# 2. At cutover: maintenance mode on the old server, then final delta sync
5ssh [email protected] 'cd /var/www/shop && php artisan down'
6rsync -aHAX --delete [email protected]:/var/www/shop/storage/ /var/www/shop/storage/
7 
8# 3. Fresh database copy streamed directly to the new server
9ssh [email protected] 'mysqldump --single-transaction --routines --triggers shop_db' | mysql shop_db
10 
11# 4. Verify before switching DNS
12mysql -e "SELECT COUNT(*) FROM shop_db.orders;"
13ssh [email protected] "mysql -e 'SELECT COUNT(*) FROM shop_db.orders;'"
14curl -sI --resolve example.com:443:127.0.0.1 https://example.com | head -1

08Availability maths: nines, dependencies and error budgets

An availability target turns into a concrete allowance of downtime. Over a 30-day month, 99.5 percent allows about 3.6 hours, 99.9 percent about 43 minutes and 99.99 percent about 4.3 minutes, which no human on-call process can meet without automatic failover. Dependencies in series multiply: if the app needs the server, the database and a payment API, each at 99.9 percent, the whole path is roughly 99.7 percent. Redundancy in parallel raises it: two independent app servers that each fail 0.1 percent of the time are both down only about 0.0001 percent, if their failures are truly independent, which shared databases, regions or deploys often break. The gap between your SLO and 100 percent is the error budget. When it is healthy, ship features faster; when it is spent, slow down and invest in reliability. That turns reliability debates into a number both business and engineers agree on.

09Deploy strategies and expand-contract migrations

A rolling deploy replaces instances one at a time behind the load balancer: cheap, but for a while old and new versions serve traffic together. Blue-green runs a full second environment and switches traffic at once, with an instant rollback by switching back, at double the cost during the release. Canary sends a small share of traffic, say 5 percent, to the new version and widens it only if error rates and latency stay healthy. All three require that two code versions can run against the same database. The expand-contract pattern makes that possible: first expand (add the new column or table, nullable, without removing anything), deploy code that writes both and reads the new one, backfill in batches, and only in a later release contract by dropping the old column. Never rename or drop a column in the same release that stops using it. This discipline, not the deploy tool, is what makes zero-downtime releases real.

• Release 1: add nullable column, deploy code that writes old and new.
• Backfill in small batches, off-peak, with progress logged.
• Release 2: read only the new column. Release 3: drop the old one.

10Alerting on what users feel: SLOs and burn rates

Alerts on CPU or memory produce noise: a server at 90 percent CPU that serves every request fast is fine. Page humans on symptoms instead, using the RED method for services (rate, errors, duration) and USE for resources (utilisation, saturation, errors) when diagnosing. The most robust pattern is the burn-rate alert from the Google SRE workbook: alert when the error budget is being consumed much faster than the SLO allows, checked over a long and a short window so it fires quickly for real incidents and resolves quickly after. A burn rate of 14.4 sustained for an hour consumes 2 percent of a 30-day budget, worth waking someone for. Slower burns become tickets, not pages. Keep Prometheus label cardinality under control: user IDs or full URLs as labels create millions of series and can take the monitoring system down exactly when you need it.

1groups:
2  - name: slo
3    rules:
4      - alert: ErrorBudgetFastBurn
5        expr: |
6          (sum(rate(http_requests_total{status=~"5.."}[1h]))
7            / sum(rate(http_requests_total[1h]))) > (14.4 * 0.001)
8          and
9          (sum(rate(http_requests_total{status=~"5.."}[5m]))
10            / sum(rate(http_requests_total[5m]))) > (14.4 * 0.001)
11        for: 2m
12        labels:
13          severity: page
14        annotations:
15          summary: "99.9% SLO error budget burning 14x too fast"

11Capacity planning with Little's law

Little's law says the average number of requests in a system equals arrival rate times time spent in it. At 200 requests per second and 250 ms average response time, about 50 requests are in flight at any moment, so you need at least 50 PHP-FPM workers or equivalent concurrency, plus headroom. If latency doubles because the database slows down, the same traffic needs 100 workers, which is why slow dependencies exhaust worker pools so suddenly. Plan for peaks, not averages: many products see three to ten times average traffic at peak hours or during campaigns. Keep steady-state utilisation around 50 to 70 percent so a failed node or a spike does not tip the rest over. Load-test before launches with a tool such as k6, increasing load until latency bends, and record where it bends: that number is your real capacity. Cost follows the same logic: right-size by measured peaks, and watch egress and storage, which grow silently.