Tuesday 20th November 2018

Small delays and slower than usual UI due to high database load

Tideways database server is under higher than usual load and that delays processing of data, traces are delayed by a few minutes and causes slowness in the user interface. We are looking into it.

Edit 10:51: We have identified the database load to be most likely caused by a noisy neighbor and we have spoken to our hoster to resolve the issue as soon as possible.

Edit 11:00: Backlog is sinking rapidly now as load decreases on the database master and UI latency improves. We are waiting for an update from the hoster about the cause for this problem.

Edit 11:45: After a steady phase, load is now higher again and the backlog for trace processing is increasing again. There is currently a lag of a few minutes in processing.

Edit 12:35: The second spike in the backlog has been completely processed and everything is back to normal now. We are still investigating the reason for the performance drop and looking to add capacity in the right locations to avoid them in the future.

Edit 15:00: The database master was on a machine that is already at it resource maximum. Our hoster had to shift resources around a little to upgrade CPU of the machine. This was finished about 45 minutes ago. Since then the load became more predictable. In addition we rolled out an application level change that reduces the write load on the database master by a large degree and we hope it reduces load even further.

Edit 15:20: Resource upgrade and application level changes finally fixed the performance problems and everything is back to normal now.