Quillforge statusSubscribe

Resolved · critical incident

Docs CDN 503s in EU

Affected: Docs delivery (CDN) · , 10:05–10:27 UTC · lasted 22 min

Updates

  1. Resolved

    Resolved. Postmortem to follow.

  2. Monitoring

    Shield restarted with higher limits.

  3. Identified

    An origin shield in Frankfurt ran out of file handles.

  4. Investigating

    Docs return 503 in Europe.

Postmortem

Docs returned 503 in Europe when the origin shield ran out of file handles

Published 9 October 2026 by Mei Chen and Sam Patel

Summary

Our Frankfurt origin shield hit its open-file limit during a traffic spike from a popular docs launch. New connections failed, so EU visitors saw 503 on uncached pages.

Requests failed7.9%EU visitors, uncached pages Duration22 min Cache hit rate91%Cached pages were unaffected

Timeline

  1. docs-cdn check fails in EU-W; incident opened as Critical.

  2. File-handle exhaustion identified on the shield.

  3. Shield restarted with a higher limit.

  4. Error rate at baseline. Resolved.

Root cause

The shield's file-handle limit was the operating-system default, set when the shield served a tenth of today's traffic. Nothing alerted on handle usage.

Action items

ActionOwnerStatus
Raise file-handle limits and manage them in configDeliveryDone
Alert at 70% handle usageDeliveryDone
Second EU shield with automatic failoverSam PatelDone