Time Is a Bug: Timezones, Ordering, and Clock Drift


The analytics dashboard shows a spike in orders at midnight. Investigation reveals it’s an artifact: orders placed at 23:30 in UTC-5 are being recorded as next-day UTC. No data is wrong - the system just doesn’t agree on what “day” means.

This is a benign version of the time bug. The dangerous version: a financial system that calculates interest on “today’s date” disagrees with the user’s local date by one day, charging interest a day early. Or an event ordering system that uses server timestamps, but two servers have clocks that diverge by 500ms, and events that should be sequential appear reversed.

Time in software is more complex than it looks because there isn’t one “time” - there’s wall clock time, UTC, local time, and the assumption that they all mean the same thing.

Bug Class 1: Timezone Confusion

The root cause: storing a timestamp without storing which timezone it’s in. 2026-07-20 14:00:00 is not a point in time. It’s a local time in some unspecified location. Add the timezone and it becomes unambiguous.

The rule: store everything as UTC. Convert to the user’s local time only for display.

# Wrong: datetime without timezone is incomplete
event.timestamp = datetime.now()  # What timezone is "now"?

# Right: always UTC, always timezone-aware
from datetime import datetime, timezone
event.timestamp = datetime.now(timezone.utc)

In Postgres, use TIMESTAMPTZ (timestamp with time zone), not TIMESTAMP. The distinction matters: TIMESTAMP stores the literal values you give it with no timezone conversion. TIMESTAMPTZ converts everything to UTC on input and converts back to the session timezone on output. If you store 2026-07-20 14:00:00 as a TIMESTAMP, it’s stored exactly as written with no timezone information. If you store it as TIMESTAMPTZ with a session in UTC-5, the database stores 2026-07-20 19:00:00 UTC.

-- Use this
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()

-- Not this (timezone information is lost)
created_at TIMESTAMP NOT NULL DEFAULT NOW()

The display conversion is the responsibility of the frontend or the API response serializer:

def serialize_event(event, user_timezone):
    return {
        'id': event.id,
        # Store UTC, display local - do the conversion here, not at storage time
        'created_at': event.timestamp.astimezone(user_timezone).isoformat(),
    }

Where this breaks in practice: “run this job at midnight” - midnight where? “Show events from today” - today in which timezone? “The contract says payment is due on July 31” - July 31 ends at different times in different parts of the world. Any feature that involves “date” rather than “timestamp” requires explicit timezone handling, because a date is only meaningful relative to a timezone.

Bug Class 2: Clock Skew in Distributed Systems

In a distributed system, each server has its own clock. Two servers can disagree on “what time is it now” by hundreds of milliseconds, even with NTP synchronization. Network Time Protocol keeps clocks close to accurate, but “close” means within tens to hundreds of milliseconds, not zero.

When you use wall-clock timestamps to order events across servers, clock skew can reverse the order. Service A generates event at T+0. Service B generates an event at T+50ms. Service B’s clock is 100ms behind. Service B’s event has timestamp T-50ms. When you sort by timestamp, B’s event appears to have happened before A’s, even though it happened 50ms after.

This matters for:

  • Event sourcing systems where replay depends on order
  • Audit logs where causality must be preserved
  • Any system where “event X caused event Y” is encoded in timestamp ordering

If you need guaranteed event ordering across distributed services, don’t use wall-clock timestamps for the ordering. Options:

Monotonic sequence numbers from a single source. A database sequence or a centralized counter assigns a globally ordered number to each event. The ordering authority is a single writer - no clock comparison needed.

Logical clocks (Lamport timestamps). Each message carries a counter. When you receive a message, you advance your counter to max(local, received) + 1. This preserves causal ordering without requiring synchronized clocks.

Vector clocks. Each service maintains a counter per service. More complex but captures concurrent events correctly.

For most applications, the practical answer is: use a monotonic sequence number from the database for event ordering, and use wall-clock timestamps only for “when did this happen” display - not for ordering logic.

Bug Class 3: Daylight Saving Time

Any recurring job scheduled in local time hits the DST boundary twice a year.

A cron job scheduled to run at 2:00 AM “every day”: on the day clocks spring forward, 2:00 AM doesn’t exist in the local timezone - the clocks go from 1:59 AM directly to 3:00 AM. The job doesn’t run.

On the day clocks fall back, 2:00 AM happens twice. The job may run twice.

The fix is simple: schedule in UTC. Jobs scheduled in UTC don’t have DST transitions - UTC has no DST.

# Bad: local time, ambiguous on DST transitions
0 2 * * * /usr/local/bin/daily-report

# Good: UTC, always unambiguous
0 7 * * * /usr/local/bin/daily-report  # 7 UTC = 2 AM Eastern during EST

The same applies to anything that fires on a schedule: database scheduled events, background job schedulers, cloud function triggers. If it runs on a clock and the clock is local time, you have a DST problem waiting to happen.

For user-facing scheduled events (“remind me at 2pm tomorrow”) - the user specifies in local time, which is correct. Store the UTC equivalent. When DST transitions, recalculate.

The Quick Reference

TaskCorrect approach
Storing timestampsTIMESTAMPTZ in Postgres, UTC datetime in application code
Displaying timestampsConvert from UTC to user timezone at render time
Ordering events across servicesMonotonic sequence numbers, not wall-clock timestamps
Scheduling recurring jobsUTC timezone, always
Date arithmeticUse a proper datetime library (arrow, pendulum in Python; date-fns, Temporal in JS)
Comparing “is this today?”Convert timestamp to user’s local date before comparing

The rule that covers most cases: UTC everywhere except at the boundary with users. Store UTC, compute in UTC, schedule in UTC. Convert to local time as late as possible - at the moment you render something for a human to read.



Read more

Backend

Webhooks: Design, Security and Debugging

Webhooks are HTTP callbacks that third-party services use to notify your application of events. Building a reliable webhook receiver requires handling retries, verifying authenticity, and processing events idempotently.