Wednesday 12th April 2017

Data Collection Processing Delays

We are seeing an increasing backlog in processing monitoring and profiling data and started to investigate.

Update 17:10: The MySQL master server is under high I/O load and this affects user-interface and worker processing, creating a backlog of currently around 20 minutes. We are in contact with the hoster to identify the problem.

Update 17:43: Database master is responsive again and the backlog is being processed.

Update 17:47: Hoster informed us that there is a problem with the hardware node and they are looking in identifying the problem.

Update 17:57: While we don't see any issues anymore, we are continuing to watch this issue and wait for a notification by the hoster.

Update 18:24: Hoster identifed a process causing very high I/O wait that is now throttled and fixed.