Issue: At approximately 7:29 AM CST this morning, we experienced an unexpected temporary power outage in our Dallas data center, due to the winter storm passing through Dallas overnight. That data center houses one of our primary DNS servers, which unexpectedly required a disk scan as it rebooted. Disk errors were found and corrected, but the process caused nearly an hour of downtime for that DNS server. Unfortunately, our secondary DNS server in our Shreveport data center was not able to provide fail-over DNS service due to a configuration issue. Although all servers were online, only users connecting by IP or with cached DNS resolutions were able to access them during the DNS outage.

Resolution: The secondary DNS server configuration was corrected immediately, allowing service for many clients to be promptly restored. The primary DNS server was brought back online by 8:40 AM CST, restoring DNS service for everyone still affected.