This night worker jobs stopped processing the queue and data was not visible in the user interface.
The problem was caused by a failure in the supervisor that is watching stopped workers and periodically retries to start them. We monitor the health of databases and workers stop processing the queue, when they cannot read or write to the database. Beacuse this is usually just a glitch and fixed automatically a restart should put the workers back on track. This system malfunctioned yesterday and all the data kept pilling up in the queues.
We started workers again and the backlog of entries is processed now.
We are keeping you up to date with our mitigation strategy to avoid this in the future.