Wednesday 6th July 2016

High Elasticsearch Latency

We are looking into high latency from Elasticsearch affecting the use of Tideways.

Update: Elasticsearch cluster has run into high CPU load it could not recover from itself, we have began to restart and recover nodes one after the other and are seeing much better utilization. Expect everything to be running again in a few minutes.

Update 2: We identified the problem in the code, repeated execution of very long running queries would pile up and lead to a congestion that caused the high CPU load.