All systems are functional.

About this site

We are sharing planned and unplanned service disruptions of the Tideways Performance Monitoring and Profiling service here. If you are experiencing problems or want to inquire about ongoing problems please write to support@tideways.io.

Tideways
Operational

Scheduled Maintenance

Reboot of host machines at our hosting provider,

Our hosting provider is rebooting all host machines due to a security upgrade and this may affect the availability of Tideways. The reboot is scheduled for the period between 00:00-08:00 Europe/Berlin on 25st July, or 22:00-06:00 UTC on 24th-25th July. This affects none of our database servers, so we are expecting the downtime to be a few seconds only.

Past incidents

Sunday 23rd July 2017

Nothing to report

Saturday 22nd July 2017

Nothing to report

Friday 21st July 2017

Slow response times due to server migration

During the scheduled maintenance yesterday our hoster detected faulty hardware that needs to be migrated. This affects one of our worker and one of our database machines. The migration is happening now and as a result you can see increased latencies or 502/503 responses from the backend. We are sorry for the inconvenience.

Update 12:53 everything is much more stable since 20-30 minutes, but there are still increases latencies to some requests.

Update 15:27 Migration of the worker machine started.

Update 19:37 Migration of the machine took a while due to extensive log files on the disk. Everything is now up and running again. We have seen very little service disruption due to this and now everything is fine again.

Problems after scheduled maintenance

We are seeing problems with data processing after the reboot of several of our machines.

One of the queues has a very large backlog and is processed slowly. The other queue is current and fast. This is why you see roughly half of the data. We will increase processing power considerably in the next minutes and start working on getting the backlog to 0 on both queues.

Update 09:06: We have almost processed the full queue, only a few minutes left. You can see all your data now. One problem we identified is that the small history chart has a gab between 4:00 and 6:00 Europe/Berlin time. We will regenerate this data from the underlying data-source to fix the chart during the day.

Update 12:54: We triggered a re-generation of the chart history to make the 2 hour gap between 4:00 and 6:00 Europe/Berlin time available again. The data should be available in your small history charts in the next 5-15 minutes.

Thursday 20th July 2017

Reboot of host machines at our hosting provider, scheduled 2 days ago

Our database master and slave are affected by this, so that we cannot realistically work with manual failovers at the right time. Potential downtime is not expected to be longer than a few minutes.