management website down

Incident Report for NTP Pool

Resolved

Maintenance is complete and all has been well for a while.

Posted Apr 25, 2020 - 00:13 UTC

Monitoring

System is up again since a little while ago; in a few hours will be monitoring while cleaning up a bit more...

Posted Apr 24, 2020 - 21:12 UTC

Identified

During a routine upgrade one of the nodes the "central cluster" had a hiccup around 19:30 UTC. The monitoring system is paused and the management portal (and much of the website) is unavailable while the database server recovers.

The database server has three replicas, but the "orchestrator" didn't want to fail over to one of the working replicas.

Posted Apr 24, 2020 - 20:52 UTC

This incident affected: Management Portal, Public website, and DNS updates.