Missing DNS updates or the last ~60 hours

Incident Report for NTP Pool

Resolved

The last 2.5 days had another outage on the DNS updates. Since the previous problem we updated/changed some of the infrastructure that distributes the updates. There was a mistake so the service didn't start up appropriately after rebooting and another mistake that this wasn't monitored properly. Both have been fixed now.

The primary impact was that DNS records weren't updated (and "rotated") during this period, causing some servers to get more traffic than usual.

Credit to Eugen Coca of Stefan cel Mare University of Suceava in Romania for reporting the unusual traffic patterns.

The monitoring has now been fixed so the configuration distribution servers are checked every minute, loud alerts fire when the DNS data is out of date and more.

Posted Jul 13, 2018 - 03:10 UTC

This incident affected: DNS updates.