Summary
Elevated write latency on an OpenSearch node caused a backlog in our data collection queues, leading to memory shortages and a short gap in incoming data (13:30–13:45 UTC) as well as delayed processing of measurement and trace data afterward.
Timeline
- ~10:00 UTC — OpenSearch node write latency slowly starts increasing
- 12:15 UTC — Issue internally detected, investigation started
- 13:30–13:45 UTC — Data collection gap (no data ingested)
- 13:45 UTC — Informing our customers about known issues (queue backlog, memory shortages, possible UI errors)
- 14:15 UTC — Root cause identified: elevated write latency on an OpenSearch node; engaged hosting provider
- 15:51 UTC — Measurement data normalized; trace data still delayed ~30–45 min
- 16:10 UTC — Trace queue still catching up
- 18:11 UTC — All queues fully caught up, incident resolved
Root Cause
An OpenSearch node experienced elevated write latency, causing data collection queues to back up and exhaust available memory, which blocked further ingestion for a short window.
Impact
- ~15 minute total data gap across customers (13:30–13:45 UTC)
- Delayed (not lost) measurement and trace data for several hours afterward
- Possible transient UI errors during the incident window
Resolution
Worked with the hosting provider to resolve the underlying OpenSearch node latency issue; queues drained and fully caught up by 18:11 UTC.
Additionally, increased the number of nodes in the OpenSearch cluster, massively reducing the risk of another write-performance degradation causing similar issues in the future.
Follow-ups
- Optimizing write operations in the ingest pipeline to reduce the impact of similar performance degradations
- Improving alerting configuration to catch these issues earlier