SRE
ReliableAIOps.com IGNITE SRE Learn from Zero · Master Production · Crack Interviews
Take a Quiz
Lesson 4 of 7 · Real-World Production Operations

SRE Incident Handling, Challenges & Goals

What actually happens when a website crashes at 2:00 AM? Learn the exact 5-step incident response workflow, the SEV-1 to SEV-3 triage matrix, the top 6 production challenges, and how SRE teams measure success.

TOPIC 1 // SRE INCIDENT HANDLING

The 5 Steps of Handling a Production Incident & Severity Matrix

THE #1 RULE OF INCIDENTS: Restore the website for users FIRST; investigate the exact code bug LATER. If a new code release went out 10 minutes before the site broke, roll it back immediately before reading stack traces!
Step Action What You Do in Real Life
1. Detect Monitoring catches the issue Prometheus / Datadog sees HTTP 500 errors spike and sends a PagerDuty alert to the on-call SRE.
2. Triage Check how bad it is (Severity) Join the incident Slack channel. Assign an Incident Commander (IC). Declare SEV-1, SEV-2, or SEV-3.
3. Mitigate Stop user pain fast Roll back the latest deployment, drain traffic from a broken zone, or scale up healthy pods.
4. Resolve Confirm health returns to normal Verify error rate is back below 0.01% on Grafana dashboards and update the customer Status Page.
5. Postmortem Prevent it from happening again Write a Blameless Postmortem within 48 hours listing what broke and how to automate the guardrail.

Incident Severity & Response Matrix (SEV-1 to SEV-3)

Severity User Impact Criteria Ack SLA Update Cadence Required Incident Roles
SEV-1 Core checkout/login down, data loss risk, or >20% user requests failing. < 5 mins Every 15 mins Incident Commander (IC), Comms Lead, Scribe, Subject Matter Experts
SEV-2 Major feature degraded, SLO fast-burn triggered, partial workaround exists. < 15 mins Every 30 mins On-Call SRE (IC) + Service Owner
SEV-3 Minor component bug, non-critical background job delayed, slow-burn alert. < 2 hours Daily / Ticket Service Engineer (Business hours)

Blameless Postmortem Markdown Template (1-Click Copy)

# INCIDENT POSTMORTEM: [INC-2026-XXXX] Checkout Page HTTP 503 Outage
- **Date / Severity:** YYYY-MM-DD | SEV-1 | Total Duration: 18 minutes
- **Incident Commander:** @oncall-sre | **Status:** Action Items Assigned

## 1. Executive Summary & User Impact
Between 10:00 and 10:18 UTC, 22% of users could not complete checkout after release v1.4 was deployed. Consumed 41% of monthly 99.9% SLO error budget.

## 2. Root Cause & Trigger
- **Trigger:** Canary release v1.4 introduced an unindexed query on the `orders` table.
- **Root Cause:** Staging database had 10k rows whereas Production had 120M rows; PostgreSQL connection pool saturated within 3 minutes.

## 3. Detection & Mitigation Timeline (UTC)
- 10:01 — Prometheus `SLOHighBurnRate` alert paged Primary On-Call SRE via PagerDuty.
- 10:04 — IC declared SEV-1, verified database connection saturation in Grafana.
- 10:09 — Rolled back release v1.4 to v1.3 using `kubectl rollout undo`.
- 10:18 — HTTP 5xx dropped to 0%; incident resolved.

## 4. Corrective Action Items (Prevent Recurrence)
- [ ] [P0] Add automated canary rollback when HTTP 5xx exceeds 1% over 2 mins (@platform-sre)
- [ ] [P1] Enforce `EXPLAIN ANALYZE` check in CI for migrations on tables >10M rows (@db-sre)
PRODUCTION RUNBOOKS // FIRST 5 MINUTES

4 On-Call Troubleshooting Playbooks (Decision Trees)

HOW TO USE THESE PLAYBOOKS: When paged on-call, don't guess randomly. Follow the exact Check 1 → Check 2 → Fix path for your symptom below.

Playbook 1: HTTP 502 / 503 Bad Gateway Spike

  • Check 1: Did a deploy or config change happen in the last 15 mins? → kubectl rollout undo immediately.
  • Check 2: Are backend pods healthy? → Run kubectl get pods. Look for CrashLoopBackOff or 0/1 Ready.
  • Check 3: Did upstream database or DNS fail? → Check kubectl logs for connection timeouts.

Playbook 2: Sudden High Latency (Site is Slow)

  • Check 1: Did traffic volume 5x suddenly (bot attack or flash sale)? → Enable CDN/WAF rate limiting & scale HPA pods.
  • Check 2: Is CPU throttled or Memory swapping? → Run kubectl top pods and top.
  • Check 3: Is a database query locking rows? → Check slow query logs / active DB locks and kill stuck queries.

Playbook 3: Pod Restarting (OOMKilled / Exit 137)

  • Check 1: Confirm OOMKill with kubectl describe pod <name> | grep -i oom.
  • Check 2: Mitigation: Increase resources.limits.memory by 50% and redeploy to stabilize production.
  • Check 3: Permanent Fix: Capture heap profile (pprof/JVM dump) to fix the memory leak in code.

Playbook 4: "No Space Left on Device" Alert

  • Check 1: Run df -h (block storage) AND df -i (inodes).
  • Check 2: Find largest folders: du -sh /var/log/* | sort -hr | head -n 5.
  • Check 3: Truncate huge unrotated log file safely (truncate -s 0 app.log) and add logrotate.
TOPIC 2 // SRE CHALLENGES

Top 6 Real-World SRE Challenges (And Exact Engineering Fixes)

1. On-Call Alert Fatigue BURNOUT

Problem: Getting paged 20 times a day for minor CPU warnings that self-resolve in 2 minutes.

Fix: Only wake up an engineer if real users are experiencing errors or slow pages (SLO burn rate). Route minor warnings to daytime tickets.

2. High Cloud & Monitoring Bills FINOPS

Problem: Leaving oversized cloud servers idle at night or storing terabytes of noisy debug logs and high-cardinality metrics.

Fix: Use Kubernetes Autoscaling (HPA/Karpenter) to scale down idle nodes, and drop noisy debug logs and unbounded labels (`user_id`) before ingestion.

3. Kubernetes OOMKills & CPU Throttling CRASHES

Problem: Pods randomly restart with Exit Code 137 or suffer high p99 latency spikes under bursty traffic.

Fix: Set accurate CPU requests without overly restrictive hard CPU limits, and match Memory Requests == Memory Limits for critical services.

4. Cascading Microservice Failures OUTAGE

Problem: One slow downstream payment service locks up worker threads across 15 upstream services.

Fix: Enforce strict client timeouts (e.g., 2 seconds), exponential backoff with jitter, and circuit breakers.

5. Configuration Drift & Manual "ClickOps" DRIFT

Problem: Hotfixes clicked manually in the AWS/Azure/GCP console get overwritten during the next Terraform deployment.

Fix: Lock production cloud consoles to read-only for humans; route all changes through Git and Terraform/ArgoCD.

6. Blame Culture After Outages CULTURE

Problem: Engineers hide mistakes or fear deploying on Thursdays because people get blamed when things break.

Fix: Run strictly Blameless Postmortems that fix the CI/CD pipeline guardrails rather than blaming individuals.

TOPIC 3 // SRE GOALS & DORA METRICS

SRE Goals: How an SRE Team Measures Success

1. Reliability & Incident KPIs

  • MTTD (Mean Time to Detect): Target < 3 mins via synthetic & SLO alerts.
  • MTTR (Mean Time to Recover): Target < 20 mins via automated rollback.
  • SLO Compliance: ≥ 95% of Tier-1 services within monthly error budget.

2. DORA Delivery Velocity

  • Deployment Frequency: Multiple safe deployments per day.
  • Lead Time for Changes: < 1 hour from PR merge to production.
  • Change Failure Rate: < 5% of deployments requiring rollback.

3. Toil & Cloud Efficiency

  • Actionable Page Ratio: > 85% of pages require real human action.
  • Toil Cap: Keep manual repetitive work under 50% of SRE time.
  • Unit Cost Efficiency: Track cloud cost per 1 Million API requests.