Reliability, SRE and operations

01SLI, SLO, SLA and error budgets

A Service Level Indicator (SLI) is a measurement of user experience, such as the share of requests that succeed, or the share served in under 300 ms. A Service Level Objective (SLO) is the target for that indicator over a window: 99.9 percent of checkout requests succeed over 30 days. A Service Level Agreement (SLA) is a contract with consequences, usually refunds, and should be looser than your internal SLO so you have warning before breaking it. The error budget is the gap between perfect and the SLO: 99.9 percent allows about 43 minutes of failure per 30 days. While budget remains, ship features; when it is spent, prioritise reliability work. Measure SLIs where users feel them (at the load balancer or from synthetic checks), and alert on fast budget burn, not on every CPU spike.

1groups:
2  - name: api-slo
3    rules:
4      - alert: ApiErrorBudgetFastBurn
5        # 99.9% SLO: page if errors burn the budget 14.4x too fast
6        expr: |
7          (sum(rate(http_requests_total{job="api",status=~"5.."}[1h]))
8            / sum(rate(http_requests_total{job="api"}[1h]))) > (14.4 * 0.001)
9          and
10          (sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
11            / sum(rate(http_requests_total{job="api"}[5m]))) > (14.4 * 0.001)
12        for: 2m
13        labels:
14          severity: page
15        annotations:
16          summary: "API is burning its 30-day error budget too fast"

02On-call and incident response

On-call means someone is responsible for responding when production breaks, with a clear rotation, escalation path and enough rest. It only works if alerts are actionable: every page should mean a user-facing problem that needs a human now, with a link to the dashboard and the runbook. Everything else goes to a ticket queue. When an incident starts, follow a simple sequence. Acknowledge the alert. Assess impact and declare severity. Assign roles: an incident commander who coordinates, someone hands-on, someone communicating with users and stakeholders. Mitigate first: roll back the deploy, fail over, disable the feature flag, scale up, or block the abusive traffic. Find the root cause only after users are safe. Keep a timeline in one channel as you go, and post status updates at a regular rhythm even when nothing has changed.

• Detect, acknowledge, assess severity, assign roles.
• Mitigate before you diagnose: rollback is often the fastest fix.
• Communicate on a schedule; silence makes incidents feel worse.
• Close with verification, then schedule the postmortem.

03Runbooks and blameless postmortems

A runbook is a short, practical guide for a known situation: 'disk above 90 percent', 'queue backlog growing', 'restore the database to a point in time'. It lists symptoms, the dashboards to check, safe diagnostic commands, the fix, and when to escalate. Good runbooks let a tired engineer at 3 a.m. act correctly, and they are the first step toward automating the fix. After any significant incident, write a postmortem: what happened, the timeline, the impact, how it was detected and resolved, the contributing factors and the concrete follow-up actions with owners and dates. Keep it blameless: ask why the system allowed the mistake rather than who made it, because people hide information when they fear blame. Review follow-ups until done; a postmortem with no completed actions is just a story.

04High availability vs disaster recovery: RPO and RTO

High availability (HA) keeps a service running through the failure of a component: multiple app instances, a database replica ready for failover, redundant load balancers, health checks that route around failures. Disaster recovery (DR) is about returning after something HA cannot absorb: a region outage, data corruption, ransomware, an accidental DROP TABLE that replicates instantly to every replica. Two numbers define DR. The Recovery Point Objective (RPO) is how much data you can afford to lose, measured in time: hourly backups mean up to an hour lost. The Recovery Time Objective (RTO) is how long you can be down while restoring. Lower numbers cost more: continuous WAL archiving and point-in-time recovery for a small RPO, warm standby infrastructure for a small RTO. Keep backups off the server, encrypted and versioned, and restore them on a schedule.

• Replicas protect against hardware failure, not against bad writes; backups do.
• Follow the 3-2-1 idea: three copies, two media, one off-site.
• A restore drill measures your real RTO.

05Chaos testing and day-to-day ops

Chaos testing means injecting failures on purpose to check that the system and the team behave as expected. Start small and safe: in staging, kill an app instance, stop Redis, add latency to a payment API mock, fill a disk, and watch whether alerts fire, retries behave and the service degrades gracefully. Game days do the same with people: rehearse a database restore or a failover with the runbook in hand. Day-to-day operations is less dramatic but matters more: reviewing dashboards and alerts, applying security updates, rotating certificates and secrets, watching disk and database growth, checking backup jobs and restore tests, cleaning logs and old builds, tuning slow queries, and keeping dependencies current. Track toil, the repetitive manual work, and automate the parts that come back every week.

06Multi-window, multi-burn-rate alerting

Burn rate is how fast you spend the error budget relative to plan: a burn rate of 1 spends exactly the budget over the SLO window, 14.4 spends a 30-day budget in about two days. The SRE Workbook recommends combining several thresholds, each checked over a long and a short window. Page when 2 percent of the monthly budget is burnt in one hour (burn rate 14.4, confirmed over 5 minutes) or 5 percent in six hours (burn rate 6, confirmed over 30 minutes). Open a ticket when 10 percent is burnt in three days (burn rate 1, confirmed over 6 hours). The long window gives significance, so short blips do not page; the short window makes the alert stop soon after the problem is fixed. Use recording rules to precompute error ratios per window so alerts stay cheap to evaluate.

1groups:
2  - name: api-slo-recordings
3    rules:
4      - record: job:slo_errors:ratio_rate5m
5        expr: sum(rate(http_requests_total{job="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m]))
6      - record: job:slo_errors:ratio_rate1h
7        expr: sum(rate(http_requests_total{job="api",status=~"5.."}[1h])) / sum(rate(http_requests_total{job="api"}[1h]))
8      - record: job:slo_errors:ratio_rate30m
9        expr: sum(rate(http_requests_total{job="api",status=~"5.."}[30m])) / sum(rate(http_requests_total{job="api"}[30m]))
10      - record: job:slo_errors:ratio_rate6h
11        expr: sum(rate(http_requests_total{job="api",status=~"5.."}[6h])) / sum(rate(http_requests_total{job="api"}[6h]))
12  # Page: (1h > 14.4*0.001 and 5m > 14.4*0.001) or (6h > 6*0.001 and 30m > 6*0.001)

07Choosing SLIs well: latency histograms and dependencies

Good SLIs are ratios of good events to valid events, measured as close to users as possible. For latency, use histograms with bucket boundaries at your thresholds, then define the SLI as the share of requests faster than, say, 300 ms; percentiles averaged across instances are mathematically wrong, while bucket counts add up correctly. Exclude what is not your failure (client 4xx errors, health checks) but be strict about what is: a 200 response with an error page is a failure. Consider dependencies: if your service synchronously needs three others at 99.9 percent each, your achievable availability is lower than 99.9 percent unless you add redundancy, caching or graceful degradation. Your SLO must be weaker than the product of your hard dependencies, or you need to remove some of them from the critical path.

• SLI = good events / valid events, over a rolling window.
• Latency SLI from histogram buckets, not averaged percentiles.
• Hard dependencies cap your achievable SLO.

08Safe change: progressive delivery and fast rollback

Most incidents are caused by changes: deploys, configuration updates, migrations, certificate or DNS edits. Reduce the blast radius of each change. Deploy progressively: one instance or a small percentage of traffic first (a canary), compare its error rate and latency with the rest, then expand. Separate deploying code from releasing features with flags, so a bad feature is switched off without a redeploy. Make database migrations backward compatible so the previous app version still works after the schema changes, which keeps rollback possible. Automate rollback on canary failure where you can. Freeze risky changes around peak business events. Track change failure rate and time to restore; they tell you whether delivery is getting safer, and most reliability gains come from here rather than from more hardware.

09Backups that restore: PITR, immutability and drills

For PostgreSQL, point-in-time recovery combines a periodic base backup with continuous archiving of WAL segments, so you can restore to any moment, for example one minute before an accidental DELETE. Tools such as pgBackRest and WAL-G handle compression, encryption, retention and parallel restore to object storage. Protect backups from the same threats as production: use separate credentials that the application servers do not hold, enable object versioning or object lock so ransomware or a compromised server cannot delete them, and copy to a second region or provider. Then measure. A scheduled drill restores the latest backup onto a fresh server, runs integrity checks and a few application queries, and records how long it took. That duration is your real RTO; the age of the newest recoverable data is your real RPO.

1# pgBackRest: full backup weekly, differential daily, WAL archived continuously
2pgbackrest --stanza=main --type=full backup
3pgbackrest --stanza=main --type=diff backup
4pgbackrest --stanza=main info
5 
6# Restore to just before a bad DELETE (run on a fresh server)
7pgbackrest --stanza=main --delta \
8  --type=time "--target=2026-11-05 14:31:00+00" \
9  --target-action=promote restore