Monitoring, alerting and backups

01Metrics, logs and traces: three views of one system

Metrics are numbers over time: requests per second, error rate, response time, CPU, free disk, queue length. They are cheap to store and perfect for dashboards and alerts, but they tell you that something is wrong, not exactly what. Logs are timestamped events with detail: this request, this user, this exception and stack trace. They answer what happened, but are expensive to search at scale, so structure them as JSON with consistent fields. Traces follow one request across services: the API call, the database queries, the call to the payment provider, each with its duration, so you can see where the time went. OpenTelemetry is the vendor-neutral standard for producing all three. Start with metrics and logs; add tracing when you have several services talking to each other.

• Metrics: is it healthy? (aggregated numbers)
• Logs: what exactly happened? (individual events)
• Traces: where did this request spend its time? (one request, many hops)

02Prometheus, Grafana and Loki

Prometheus is a time-series database that pulls (scrapes) metrics over HTTP from targets every few seconds. Exporters expose metrics for things you did not write: node_exporter for CPU, memory and disk, postgres_exporter for the database, and your own app can expose a /metrics endpoint. You query it with PromQL and define alert rules; Alertmanager then groups, deduplicates and routes those alerts to email, Slack or a pager. Grafana draws dashboards on top of Prometheus and many other sources. Loki stores logs cheaply by indexing only labels such as app, environment and level, not the full text, and Grafana Alloy is the collector that ships logs and metrics to it. Together they give a self-hosted stack that runs comfortably beside a Docker Compose application.

1groups:
2  - name: host
3    rules:
4      - alert: DiskAlmostFull
5        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.15
6        for: 15m
7        labels: { severity: warning }
8        annotations:
9          summary: "Less than 15% disk left on {{ $labels.instance }}"
10      - alert: HighErrorRate
11        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
12        for: 10m
13        labels: { severity: critical }

03Uptime checks, error tracking and alerts that do not spam

An uptime check probes your site from outside, every minute, the way a user would: does the homepage return 200, is the TLS certificate valid for more than two weeks, does the health endpoint confirm the database answers. Tools like Uptime Kuma can be self-hosted; hosted checkers test from several regions. Error tracking (Sentry for web and backend, Firebase Crashlytics for mobile) groups exceptions and crashes, shows the stack trace, release and affected users. For alerts, page a human only for symptoms users feel (site down, error rate high, checkout failing) and send causes like high CPU to a dashboard or a low-priority channel. Require a condition to last several minutes before firing, group related alerts, and every alert should link to a runbook explaining what to check first.

• Alert on symptoms, investigate causes.
• Every page must be actionable; if nobody needs to act, it is not a page.
• Review noisy alerts monthly and delete or tune them.

04Backups: the 3-2-1 rule, snapshots vs real backups

The 3-2-1 rule: keep three copies of your data, on two different types of storage, with one copy off-site. A modern extension adds one copy that is immutable or offline, so ransomware or a stolen admin key cannot delete it, and zero errors after a verified restore. A provider snapshot copies the whole disk at a point in time. It is fast to restore a whole server, but it usually lives in the same provider account, and a snapshot of a running database can be inconsistent. A real backup is a consistent, application-level copy, such as pg_dump or mysqldump output plus uploaded files, stored somewhere else and encrypted. Use both: snapshots for fast full-server recovery, logical backups for precise, portable restores of data.

1#!/usr/bin/env bash
2set -euo pipefail
3STAMP=$(date +%F_%H%M)
4pg_dump -Fc -U app appdb > /backups/appdb_$STAMP.dump
5 
6# encrypted, deduplicated, off-site copy (S3-compatible storage)
7export RESTIC_REPOSITORY=s3:https://s3.example.com/your-backup-bucket
8restic backup /backups /var/www/app/shared/storage
9restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
10restic check

05Testing restores, RPO and RTO

Two numbers define a backup strategy, and they should be agreed with the business, not guessed by engineers. RPO, recovery point objective, is how much data you can afford to lose, measured in time: with nightly dumps your RPO is up to 24 hours; with continuous WAL archiving or binlog shipping it can be minutes. RTO, recovery time objective, is how long you can be down while restoring. Then prove you can meet them. On a schedule, restore the latest backup into a scratch database or a temporary server, run a few sanity queries, check uploaded files open, and time the whole process. Write the steps down as a runbook so anyone on the team can do it under stress.

1# monthly restore drill into a throwaway database
2createdb restore_test
3pg_restore --no-owner -d restore_test /backups/appdb_latest.dump
4psql -d restore_test -c "SELECT count(*), max(created_at) FROM orders;"
5dropdb restore_test

06SLIs, SLOs, error budgets and burn-rate alerts

A service level indicator (SLI) is a measured ratio of good events to all events, for example the share of requests answered successfully in under 500 ms. A service level objective (SLO) is the target for it over a window, say 99.9% over 30 days. The remaining 0.1% is the error budget: about 43 minutes of full outage per month, or a longer period of partial errors. Instead of alerting on a fixed error rate, alert on how fast the budget is burning. A burn rate of 1 uses exactly the budget in 30 days; a burn rate of 14.4 sustained for an hour consumes 2% of the monthly budget and deserves a page. The multiwindow pattern from Google's SRE workbook checks a long window to be sure the problem is real and a short window to be sure it is still happening, so alerts fire fast and clear fast.

1# page: 2% of a 30-day 99.9% budget burned in 1 hour
2- alert: ErrorBudgetFastBurn
3  expr: |
4    (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4 * 0.001)
5    and
6    (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4 * 0.001)
7  labels: { severity: page }
8  annotations: { runbook: "https://wiki.example.com/runbooks/error-budget" }

07Prometheus internals: series, cardinality and histograms

Prometheus stores one time series for every unique combination of metric name and label values. http_requests_total with labels for method, route and status might produce a few hundred series, which is fine. Add a user_id or full URL label and you create one series per user or per URL, memory explodes and queries crawl; this is the cardinality problem, and it is the most common way to break Prometheus. Keep labels to small, bounded sets and put high-cardinality detail into logs or traces. For latency, use histograms: the app counts requests into buckets, and histogram_quantile estimates p95 or p99 at query time, across all instances. Averages hide the slow tail that users actually feel. Expensive queries used on dashboards and alerts should become recording rules, computed once per interval. Prometheus keeps 15 days by default; long-term storage usually goes to Thanos, Mimir or a hosted backend via remote_write.

1# p95 latency per route over 5 minutes
2histogram_quantile(0.95,
3  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
4 
5# recording rule: compute once, reuse in dashboards and alerts
6- record: route:http_request_duration_seconds:p95_5m
7  expr: histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
8 
9# find the metrics with the most series
10topk(10, count by (__name__) ({__name__=~".+"}))

08Point-in-time recovery for PostgreSQL

A nightly dump gives an RPO of up to a day. Point-in-time recovery (PITR) brings it down to minutes. PostgreSQL writes every change to its write-ahead log (WAL) before applying it. If you keep a periodic base backup, a physical copy of the data directory, plus every WAL segment since then, you can restore the base backup and replay the WAL up to any moment, for example one minute before someone ran a DELETE without a WHERE clause. You enable it with archive_mode and an archive command that ships each finished segment off the server. In practice use a dedicated tool such as pgBackRest or WAL-G, which handle compression, encryption, retention, parallel restore and verification for you, and push to object storage. Restore by placing the base backup, setting recovery_target_time and creating a recovery.signal file. Managed databases offer the same feature as a button, but you still need to rehearse it.

1# postgresql.conf (with pgBackRest)
2wal_level = replica
3archive_mode = on
4archive_command = 'pgbackrest --stanza=app archive-push %p'
5archive_timeout = 60          # ship at least once a minute
6 
7# restore to just before the mistake
8pgbackrest --stanza=app --type=time \
9  --target="2026-11-05 14:31:00+00" --target-action=promote restore

09Protecting the backups themselves

Modern ransomware and malicious insiders go for the backups first, so assume an attacker with the server's credentials will try to delete them. Three defences work together. Separate credentials: the server should be able to add new backups but not delete old ones. restic's rest-server has an append-only mode, and object storage can grant write without delete. Immutability: S3 Object Lock (also offered by several S3-compatible providers) stores objects in write-once mode for a retention period; in compliance mode nobody, not even the account root, can shorten it. Independent monitoring: a dead man's switch expects a ping after every successful backup and alerts when one is missing, which catches silently broken cron jobs. Keep the encryption key or password somewhere separate from the server, such as a password manager, because an encrypted backup without its key is just noise.

1# bucket must have versioning; lock new objects for 30 days
2aws s3api put-object-lock-configuration --bucket example-backups \
3  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'
4 
5# end of backup script: report success to a dead man's switch
6restic backup /backups && curl -fsS -m 10 --retry 3 https://hc-ping.example.com/your-check-id