Production is on fire in development — what it is, causes, and action plan

What “production is on fire” means and types of failures

The phrase “production is on fire” (everything is down) describes a situation where the production environment is malfunctioning and users are affected. The failure may manifest as complete app unavailability (blank screen, 502 error), partial unavailability (payment module not working but other functions accessible), or performance degradation (extremely slow loading). Incident severity is determined by the percentage of affected users and the duration of the outage.

According to Atlassian Statuspage (2025), the average downtime for mobile applications in 2024 was 27 minutes per incident. The most common causes: code regression after deployment (34%), cloud provider outage (22%), database issues (18%), configuration errors (15%), and DDoS attacks (11%). Key takeaway: most outages are caused by changes the team made themselves, not by external factors.

It is important to distinguish between a crash (app failure on the client side) and a backend outage (server unavailability). A crash is usually fixed with a client-side hotfix, while a backend outage requires infrastructure changes or service redeployment. Monitoring metrics: for the client — crash-free rate, for the server — 5xx error rate and p95 latency. APM (Application Performance Monitoring) — Sentry, New Relic, Datadog — helps quickly determine the type of failure.

Incident severity: P0, P1, P2 and classification criteria

A unified severity classification is the foundation of a fast response. Without it, the team wastes time discussing “how urgent is this” instead of taking action. The classic scale: P0 (critical) — the application is completely unavailable or user data is leaking, response time — immediate; P1 (high) — critical functionality is broken for 50%+ of users, response time — 15 minutes; P2 (medium) — non-critical functionality is unavailable for some users, response time — 1 hour.

P0 requires immediate escalation: the on-call engineer interrupts any current work and switches to the incident. If the problem is not resolved within 10 minutes — the tech lead is brought in. If after 30 minutes — escalation to the engineering manager. For P0 incidents, it is acceptable to break any processes: make a hotfix without a full code review, deploy directly to production, ignore branch protection rules. Emergency override must be pre-agreed at the team level.

Severity table

It is extremely important not to underestimate the severity. P0 and P1 incidents classified as P2 lead to delayed response and increased downtime. Rule: if in doubt — set P0. Over-classification is better than under-classification: it is better to hold an extra meeting than to lose an hour of recovery time.

First 10 minutes: action algorithm during an outage

Timer starts: from the moment an alert or user report comes in. The first 10 minutes are the most critical. Algorithm: 1) confirm the issue — make sure the problem is real (not a false alarm); 2) stop the bleeding — immediately reduce impact (rollback, feature toggle, endpoint blocking); 3) communicate — post in the general #incident channel: what happened, severity, and what is being done. The first 10 minutes are not spent on root cause analysis.

In parallel with stopping the bleeding, one engineer starts diagnosis while another handles communication. Communication channels: Slack #incident channel (for the team), status page (for users), email/SMS escalation (for management). Every 15 minutes — a status update with information: what is known, what is being done, and estimated recovery time. Status page (StatusPage, Statuspal) displays uptime and incident history for external users.

How to stop the bleeding: rollback, feature toggle, and hotfix

The first and most important rule: do not try to fix the problem on production. If a new release caused the failure — roll back to the previous stable version. If the failure is caused by a specific feature that is behind a feature toggle — simply disable the toggle. If neither rollback nor toggle is available — apply a hotfix with a minimal diff. Rollback is the safest option because it returns to a state that was already working.

Feature toggle (aka feature flag) is a powerful tool for stop-the-bleeding without a deployment. If the payment module crashes but is disabled via toggle — users simply do not see the payment button instead of getting an error screen. A toggle does not require a build, does not require a store review, and takes effect in seconds. Every critical feature should be behind a feature toggle with remote config capability. Feature flag — the first line of defense.

If rollback is impossible (e.g., due to an irreversible database migration) and no toggle is in place — the last resort is a hotfix with minimal changes. The hotfix is created from the latest release tag, contains only the lines necessary to fix the issue, and goes through fast-track deployment (see the article “Hotfix — urgent fixes”). Golden rule: after stabilization, always conduct a root cause analysis, even if the cause seems obvious.

Diagnosing causes: logs, metrics, and alerts

After stopping the bleeding (or in parallel, if enough engineers are available), diagnosis begins. The first source is logs. Centralized logging (ELK, Grafana Loki, Datadog Logs) allows finding errors by timestamp, user ID, or request ID. Important: logs must be structured (JSON) for fast grep. Structured logging is a mandatory requirement for all services.

The second source is metrics. Grafana, Datadog, New Relic show when the error spike occurred, on which endpoints, and with which status codes. Comparing metrics before and after deployment helps localize the problem to a specific service or endpoint. RED metrics (Rate, Errors, Duration) — the standard for microservice monitoring.

The third source is distributed tracing. Jaeger, Zipkin, Datadog APM show the request path through microservices and identify where exactly the delay or error occurred. Tracing is especially useful for cascading failures, when a failure in one service causes errors in all dependent services. Trace ID must be passed from the client to all backend services.

# Quick diagnostic example using kubectl and logs
# List pods with errors
kubectl get pods --field-selector=status.phase!=Running

# Check logs of crashed pod
kubectl logs --previous pod/auth-service-7f4b9c5d6-abc12

# Search errors in service for the last 30 minutes
kubectl logs deployment/api-gateway --since=30m
  | grep "5[0-9][0-9]" | head -50

Important: do not attempt to diagnose the cause before stopping the bleeding. If 50% of users are seeing a crash — roll back first, then investigate. Exception: if a rollback would take longer than a direct hotfix (e.g., due to data incompatibility). In this case, apply the hotfix immediately and conduct the post-mortem after stabilization. Diagnosis before fix is a dangerous pattern that increases downtime.

Post-mortem: how to review incidents without blaming

Post-mortem (also called incident review) is a structured incident analysis conducted 24–72 hours after resolution. Its purpose: to understand why the failure occurred, why monitoring and tests did not catch it before production, and what to change in processes to prevent recurrence. Blameless culture is a fundamental principle: the post-mortem discusses processes, tools, and communication, not the mistakes of specific individuals.

Structure of a post-mortem document: timeline (chronology of events with timestamps), impact (affected users, duration, financial losses), root cause (technical root cause), detection (how it was discovered, why it was not caught earlier), response (what was done, what could have been done faster), action items (specific tasks with assignees and deadlines). Action items must be S.M.A.R.T.: specific, measurable, assignable, realistic, time-bound.

Typical action items after a production outage: add monitoring and alerts for the metric that was silent; expand test coverage for the missed case; add a page to the runbook with a step-by-step algorithm for a similar situation; conduct team training on the tool that was used incorrectly. Every action item is a concrete change that reduces the likelihood of the incident recurring.