Between 14:00 and 20:00 UTC, congestion in our internal log drain was causing widespread latency bursts, causing intermittent and systemic failures in user-facing requests. The cascade of buffer overflows and request queues propagated right up to our frontend servers several times.
We've provisioned larger and more reliable capacity for our log drains, and added monitoring of its throughput.
Posted Feb 12, 2016 - 10:37 AEDT
Investigating
We’re currently aware of an issue affecting our users. We’re doing our best to resolve this ASAP and will provide updates.