Friday 23rd November 2018

Large ingest due to Black Friday leads to small delay in processing

Monitoring data is delayed in processing roughly 2-3 minutes, because we haven't fully accounted for the massive increase in load due to Black Friday. As our primary database master looks to be the bottleneck, we are looking how to carefully increase throughput.

Edit 16:00 We rolled out a new version that doesn't update database rows for each server and operation whenever new data comes in. We have been seeing very slow UPDATEs in the worker processes that change "last_recorded" timestamp information that aren't necessary in the high ratios that they were previously executed with.