Summary

A maintenance query to delete old entries in a large table was not adequately optimized and required a lot of load. This brought the primary database server under heavy load. Cancelling the query lead to a rollback that also took a while for the system to recover finally.

Customers experienced a dataloss of incoming data between 13:42 and 14:28 UTC time on October 16th, 2026.

We are very sorry for the inconvenience and you can be assured that we are following up on this incident to make sure a similar mistake does not happen again and is otherwise softened by better automatic recovery and retry mechanisms.

Timeline

• 13:30 UTC - Maintenance query to large table was started.
• 13:42 UTC - Database server load increased to a level that made UI/Ingest unavailable.
• 13:49 UTC - query was killed and started to roll back.
• 14:28 UTC - system recovered and load normalized.

Root Cause

An unoptimized maintenance SQL query to cleanup a large table caused high load.

Impact

• ~45 minute total data gap across customers (13:42–14:28 UTC)
• Possible transient UI errors during the incident window

Follow-ups

• Current defined best-practices from Ops team were not followed for the maintenance query, so work is being done on reviewing them in the team and potentially surfacing them during code-reviews automatically.
• We are investigating how the retry mechanism of daemons from customer machines worked and if we can optimize and improve it for the future.