Postmortem: The March 2025 Geocoding Latency Incident

On March 14, 2025 at 14:47 UTC, our geocoding service started returning 95th percentile latencies of 2.3 seconds instead of our normal 150ms. The incident lasted 47 minutes and affected approximately 12% of our requests during that window. This is our postmortem.

Timeline

14:47 UTC: Monitoring alerts fire. Geocoding latency spikes to 1.2 seconds.
14:49 UTC: On-call engineer pages the team. Initial hypothesis: database load.
14:53 UTC: We check database metrics. CPU and disk I/O normal. Hypothesis shifts to network.
15:02 UTC: Network team confirms a fiber cut in our primary data center uplink affecting 30% of egress capacity.
15:15 UTC: We begin draining traffic from the affected data center to secondaries.
15:34 UTC: Traffic is mostly drained. Latencies begin dropping.
15:45 UTC: Latencies return to baseline. We declare the incident resolved.

Root Cause

A fiber optic cable in our primary data center suffered physical damage. Later investigation determined that construction equipment struck it while laying new infrastructure. This reduced our primary data center's external connectivity by 30%, forcing traffic to route through secondary paths with higher latency. Our geocoding service makes outbound calls to our geocoding database cluster, so the reduced bandwidth directly impacted response times.

Why wasn't this automatically handled? Our traffic balancing logic assumed that if a data center was still reachable, it should remain active. We didn't have an automated circuit breaker based on egress capacity. This was a gap in our resilience design.

Why It Lasted 47 Minutes

We didn't immediately drain the data center because our monitoring didn't directly surface the fiber cut. We knew latency was high, but we didn't know why. It took 15 minutes to correlate the latency spike with the network team's reports of reduced uplink capacity. Manual intervention then took another 19 minutes. This time lag was frustrating, and we've fixed it.

What We Fixed

We've implemented three changes to prevent recurrence:

Network-aware load balancing: Our traffic router now monitors egress capacity as a primary signal. If a data center's available bandwidth drops below 50% of baseline, we begin draining traffic automatically.
Cross-functional alerting: When network capacity degrades, our network team is automatically notified and engaged, not just our application team. We've integrated our monitoring with their on-call escalation.
Better chaos testing: We now regularly simulate network degradation in staging environments to validate our response. We run a monthly chaos drill.

Impact

We estimate the incident cost affected customers approximately $280 in failed or slow requests (we've issued credits to all affected accounts). More importantly, we fell short of our reliability promise. We're committed to not letting this happen again.

What Didn't Fail

Some things worked well: our monitoring caught the issue immediately, our runbooks were clear enough to guide the on-call engineer quickly, and our secondary data centers handled the failover gracefully without errors. Our customers who implemented proper retry logic experienced only brief slowdowns.

Lessons for Our Users

This incident reinforced something we always recommend: implement exponential backoff and read our Retry-After headers. When you encounter a slow response or 429 status, waiting a few hundred milliseconds and retrying is the right move. Many of our most reliable customers barely noticed this incident because their clients already handle transient failures well.

Transparency

We're sharing this postmortem because we believe reliability is built on honesty. We made mistakes, we fixed them, and we're stronger for it. If you have questions about the incident or want to discuss our infrastructure reliability practices, reach out at trust@forge-api.dev.