Wednesday 29th August 2018 not reachable through internet right now

We are seeing and as not reachable right now as it looks like our hoster is not reachable over the public internet right now. We are investigating.

21:19: Our hoster is reporting a partial outage, see their status here:

21:26: This looks like to be a network problem at SysEleven, as Tideways is still publishing metrics out to our external time series system.

21:38: Update from hoster via e-mail confirming problems with network reachability. They are working on a fix.

22:00: and are reachable again and daemons have started connecting to the backend. We are watching the status actively.

22:12: UP and DOWN status are switching over the last minutes, incident does not look to be brought to a conclusion.

23:00: SysEleven Status Page is all green again.

Friday 31st August: Our hosting company SysEleven notified customers that outage was related to a DDoS attack on their infrastracture and that they will share more information at a later stage, since investigation and analysis is still ongoing.

Post Mortem

RIght now we are waiting for additional information from the hoster on the DDoS and potential future mitigation. Independent from this we have started thinking about how to improve our data ingestion architecture to cope with network outages. Please follow this post for an update on this outage at a later point.