Traffic spikes to 3x normal. Response times climb from 200ms to 3 seconds. Requests queue up. Memory fills with queued connections. The service OOMs and goes down. Everything that was waiting also fails. Total downtime: 45 minutes.

This is the failure mode that load shedding prevents. The service didn’t crash because of a bug. It crashed because it accepted more work than it could handle and had no mechanism to say no.

A system under overload has two options: degrade gracefully, or fail catastrophically. Load shedding is choosing to degrade gracefully, deliberately.

What Load Shedding Is

Load shedding is rejecting requests you can’t handle. Instead of queueing them until memory is exhausted, you return a fast error and let the client decide what to do.

The tradeoff is explicit: some requests fail, but the system stays up for the requests it can handle. Without load shedding, unbounded queuing leads to degraded performance for everyone, then total failure for everyone.

The key insight: a 503 returned in 10ms is better for the overall system than a 200 returned in 30 seconds. The 503 frees the connection, releases the memory, and gives the client actionable information. The 30-second 200 holds the connection, accumulates queue depth, and creates the conditions for total failure.

from collections import deque

MAX_QUEUE_DEPTH = 500

class RequestHandler:
    def __init__(self):
        self.queue = deque()

    def handle(self, request):
        if len(self.queue) >= MAX_QUEUE_DEPTH:
            # Shed the load - fast rejection
            return Response(
                status=503,
                headers={'Retry-After': '5'},
                body='Service temporarily at capacity'
            )

        self.queue.append(request)
        return self._process()

Where to set the threshold: instrument your service under load tests and find the queue depth at which response times start exceeding your SLO. Set the threshold below that point. You want to start shedding before the service degrades, not after.

Backpressure

Backpressure is the mechanism by which a consumer signals upstream that it’s overwhelmed. The consumer is full - the producer should slow down or stop sending.

Without backpressure, producers send at whatever rate they can, and consumers fall further and further behind. The queue grows until something gives. With backpressure, the consumer’s capacity limits the producer.

In practice: a database connection pool with a maximum of 20 connections is backpressure. When all 20 are in use, new queries wait - or if you add a timeout, fail fast. The database’s capacity constrains how fast the application can submit queries.

A queue with a bounded size is backpressure. When the queue is full, you have to decide: drop the new item, drop the oldest item, or block the producer. Each choice is correct for different situations:

• Drop new items (tail drop): when old data is more valuable than new - a live metrics stream where missing a data point is fine, but dropping older history would be worse.
• Drop oldest items: when the most recent data is what matters - a real-time dashboard where stale metrics are useless.
• Block the producer: when you can’t lose any items and can apply backpressure upstream - a transactional system where every message must be processed.

The wrong choice in most cases: an unbounded queue. An unbounded queue that fills up is not a buffer - it’s a time-delayed crash. You’ve delayed the failure by storing the problem in memory.

Circuit Breakers

A circuit breaker prevents cascading failure when a downstream service is struggling.

Without a circuit breaker: service A calls service B. Service B starts timing out. Service A’s requests pile up waiting for B’s responses. Service A runs out of threads/connections. Service A starts failing. Service C, which calls A, starts failing. The overload propagates upstream.

With a circuit breaker: after a threshold of failures, the circuit “opens” - service A stops calling service B and immediately returns an error. Service B gets no traffic, which gives it a chance to recover. Service A returns fast errors instead of slow ones, which keeps its own thread pool available for requests to healthy services.

The three states:

Closed (normal): requests flow through. Failures are counted.

Open (failing): after the failure threshold is reached, requests immediately return an error without calling the downstream service. A timer starts.

Half-open (probing): after the timeout, one request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit stays open and the timer resets.

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30):
        self.failure_count = 0
        self.last_failure_time = None
        self.state = 'closed'
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout

    def call(self, func, *args, **kwargs):
        if self.state == 'open':
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = 'half-open'
            else:
                raise CircuitOpenError("Downstream unavailable, try later")

        try:
            result = func(*args, **kwargs)
            if self.state == 'half-open':
                self.state = 'closed'
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = 'open'
            raise

What the circuit breaker buys you: fast failure instead of slow failure. A 10ms error is better than a 10-second timeout for both the caller (which can react faster) and the callee (which gets breathing room to recover).

Priority Shedding

Not all requests are equal. When the system is under load, shedding work intelligently is better than shedding randomly.

A few ways to prioritize:

By endpoint cost. A report generation job is expensive. A health check is cheap. An authentication check is critical. When load is high, shed the expensive non-critical operations first.

By user tier. A paid user being served matters more to business continuity than a trial user. This is a business decision, not just a technical one.

By request age. Requests that have been queued for longer than your SLO are already going to fail on latency. Drop them and free the capacity for fresh requests.

def priority_shed(request, queue_depth):
    if queue_depth > HIGH_THRESHOLD:
        # Shed expensive non-critical requests
        if request.endpoint in EXPENSIVE_ENDPOINTS:
            return reject(request)
    
    if queue_depth > MEDIUM_THRESHOLD:
        # Shed requests already past their deadline
        if request.elapsed_ms > request.timeout_ms:
            return reject(request)
    
    return accept(request)

The Retry Storm

Load shedding creates a risk: if you return 503 and clients immediately retry, you’ve moved the problem without solving it. The retry itself adds to the load you’re trying to shed.

Clients should implement exponential backoff with jitter on 503 responses:

import random, time

def call_with_backoff(func, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return func()
        except ServiceUnavailable as e:
            if attempt == max_attempts - 1:
                raise
            # Exponential backoff + jitter: don't retry at exactly the same time as everyone else
            sleep_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(sleep_time)

The Retry-After response header tells clients how long to wait before retrying. Respecting it reduces coordinated retry storms.

The Capacity Baseline

None of these mechanisms work without knowing your baseline. If you don’t know what normal load looks like, you can’t set meaningful thresholds.

Run load tests before you need to. Know your service’s behavior at 1x, 2x, and 3x normal load: where do response times start degrading, where do errors start appearing, what’s the breaking point? Set your load shedding thresholds based on that data, not on intuition.

Instrument queue depths and rejection rates as first-class metrics, alongside latency and error rates. A queue depth that’s trending upward is a leading indicator of a problem. A rejection rate that spikes is the load shedding working as designed. If you can’t see these metrics, you can’t tell the difference between the system protecting itself and the system failing silently.