Tuesday 22nd August 2017

Data Collection Large backlog of data processing

We have a large backlog of items that need to be processed on one queue. Since we are randomly sharding incoming data, this means that you probably see 50% current data and empty data in between where the data is coming from the slow shard. We are seeing to it that the lagging queue is completely processed as soon as possible.

Edit 08:57: The preview chart has a gap between 4:00-5:00 Europe/Berlin time that is due to a bug in the aggregation. In that time window our hosting provider performed maintenance work on our infrastructure and when this was completed the preview chart was created although there was no data recorded yet in the timeframe, since the queues where stopped for processing. When the queue backlog is fully processed we will re-generate the preview charts with the correct data.

Edit 09:38 All backlogs have been processed and data is up to date again and preview charts have been regenerated to include data between 4:00 and 5:00 Europe/Berlin time.