Slow processing of historical aggregation job

Incident Report for Tideways

Resolved

Processing will be up to date in the next 60-90 minutes, before processing for June 14th gets scheduled. This restores equilibrium in processing historical data.
Posted Jun 15, 2026 - 00:19 CEST

Monitoring

Processing queue has dropped from over 60k tasks to 39k tasks now. All historical data for June 12th have been processed and now those from June 13th are being processed as fast as possible.
Posted Jun 14, 2026 - 23:39 CEST

Identified

The issue was identified as not locking the processing for the same application/project, which lead to all workers re-processing the same project which was taking a few minutes and then breaking with an out of memory error.

Multiple fixes are now in place and the queue for historical data is processing to close the gap as soon as possible.
Posted Jun 14, 2026 - 23:13 CEST

Investigating

We are investigating slow processing of historical data aggregation job.
Posted Jun 14, 2026 - 21:14 CEST
This incident affected: Ingest / Data Collection.