Wednesday 10th July 2019

Performance Degradation, Delay in processing

Due to a currently unknown performance degradation in Tideways the processing of data is backed up by 10 minute delay that is increasing. We are investigating the problem and working on a fix.

Edit 8:36 We have identified slow database SQL update queries inside our worker jobs to cause the delays. Due to the data processing task taking around 2-4 seconds per task now instead of a few hundred milliseconds, the queue of incoming data is not processed fast enough to stay empty.

Edit 9:30 We disabled a few non critical update queries that updated timestamps for now, with the effect that queues are now processing much faster and should catch up in the next minutes. We now turn into finding out why the queries suddenly got much slower.

Edit 9:45 Backlog has been processed, all data is now current again.

Edit 17:02, 11th July The performance problem could be traced back to a hardware problem and was fixed by the hoster. Everything is back to normal now and the workers are not under pressure anymore to process data.