Production debugging is different from development debugging. You can’t add print statements and rerun. You can’t step through with a debugger. The system is live, users are affected, and every minute matters. The only information you have is what was logged before you knew there was a problem.

This article is about process. How you approach a production issue is as important as the technical tools you use.

Establish the Blast Radius First

Before you start looking for root causes, understand scope. This question drives how urgently you work and who needs to be involved.

• How many users are affected? All users, users of a specific feature, users in a specific region?
• Is the system completely down or degraded?
• Is the problem getting worse, stable, or recovering?
• When did it start?

Check your metrics dashboard first. Error rate graphs, traffic graphs, latency percentiles - these tell you scope and timeline faster than reading logs. A spike in 5xx errors starting at 14:32 UTC is more precise than “it seems slow.”

Knowing when it started is valuable because you can correlate it with deployments, configuration changes, external events (scheduled jobs, traffic spikes), and infrastructure changes.

The Three-Layer Check

For most production issues, the root cause is in one of three places:

Application layer: your code. Recent deployments, configuration changes, code bugs.

Infrastructure layer: the servers, databases, load balancers, and cloud services your code runs on. Resource exhaustion (CPU, memory, disk), network issues, database overload.

External dependencies: third-party APIs, external services, CDNs. If your payment provider is down, your checkout is down, and the root cause isn’t in your code.

Check them in order:

1. Did anything deploy recently? A deployment 5 minutes before the incident started is the most likely cause.
2. Are infrastructure metrics normal? CPU, memory, disk, network I/O.
3. Are external dependencies responding? Check their status pages.

This check takes 2-3 minutes and eliminates categories before you start reading logs.

Reading Logs Effectively

When you open the logs, you’re looking for anomalies - errors, warnings, or patterns that don’t appear in normal operation.

Start with errors. Filter to ERROR and FATAL level messages around the time the incident started. A spike in errors that started at the same time as the incident is likely the cause or a symptom of it.

Correlate with a request ID or trace ID. A single failing request in a complex system might touch 5 services. If you have distributed tracing, find a trace for a failing request and follow it through every service. Without tracing, use the request ID that you (should be) logging with every log line.

# Good: every log line includes the request ID
logger.error("Failed to process payment", extra={
    "request_id": request.id,
    "user_id": user.id,
    "error": str(e)
})

Without a correlation ID, you’re matching log lines by timestamp and hoping you’re following the right request.

Look for the first occurrence of the error, not the most recent. Log lines that say “database connection failed” might appear thousands of times, but the root cause was a single event earlier - a connection pool exhaustion, a config change, a primary failover.

The Hypothesis Method

Don’t read logs aimlessly looking for something wrong. Form a hypothesis, then look for evidence to confirm or refute it.

“Hypothesis: the incident started when we deployed at 14:30 and the new version has a bug in the payment flow.”

Evidence for: errors started at 14:32, errors are in the payment service, error message mentions PaymentProcessor.charge().

Evidence against: errors started before the deployment, errors are spread across all endpoints.

If the evidence doesn’t support your hypothesis, form a new one. This is faster than reading all the logs in hopes of finding something.

Common hypotheses worth testing first:

• Recent deployment is the cause
• Database is slow or unavailable
• An external service is failing
• A background job ran and caused resource contention
• Traffic increased beyond what the system handles

When You’re Stuck

Signs you’re going in circles: you’ve been reading logs for 30 minutes, you have a theory but no confirmation, you’re reading the same log lines again.

At this point:

• Bring in someone else. A fresh pair of eyes consistently catches things you’ve been staring past.
• Escalate to the team that owns the service if it’s not yours.
• Roll back the most recent deployment if you haven’t already. Even if you don’t think it’s the cause, a rollback buys time.
• Add more logging if necessary and wait for another failure to occur. Sometimes you genuinely don’t have enough information to diagnose without instrumenting further.

Rollback is underutilized. Many engineers resist rolling back because they’re confident the deployment isn’t the cause. But rolling back when you’re stuck costs little - if the incident continues after rollback, you’ve eliminated the deployment and can focus elsewhere. If the incident resolves, the deployment was the cause and you saved more debugging time.

Mitigation vs Root Cause

In an active incident, separate mitigation from root cause analysis. Mitigation stops the bleeding. Root cause analysis explains why.

Mitigation might be: rolling back, disabling a feature, increasing a connection pool limit, restarting a process, failing over to a backup database. You can sometimes mitigate in minutes without fully understanding the cause.

Once mitigated, the urgency drops. Root cause analysis can be thorough and deliberate, which produces better results than rushed investigation.

Document the timeline as you go - when the incident started, when you investigated what, when mitigation happened, when full recovery occurred. This feeds the postmortem and helps pattern-match with future incidents.

Before the Next Incident

The questions to ask after every production issue:

What made this hard to find? Missing logs? Unclear error messages? No distributed tracing? Each of these is an investment to make.

What would have caught this before production? A test that didn’t exist? A staging environment that didn’t replicate the failure condition? An alert threshold that was too loose?

The goal is that each incident leaves the system slightly more observable and slightly more resilient than before. Over time, this compounds.