As has been widely reported, a major DDOS attack occurred last Friday, October 21, 2016 against the DNS provider, Dyn. This attack affected thousands of Internet services such as Github, Twitter, Spotify, Reddit, and Netflix, as well as Acquia. The attack began at approximately 11:10 UTC and had several "waves" where some services were affected until Dyn declared the incident resolved at 22:17 UTC. See Dyn's status incident report for more information on their response and mitigation.
Acquia uses Dyn's services for DNS resolution on many of our domains. These included many of the sites that our customers use to manage their Acquia subscriptions as well as many back-end services such as our hosted Solr search servers.
At approximately 11:54 UTC, the effects of the DNS outage began to become apparent to our engineers and to our customers. As we realized that the scope of the problem was widespread, we began adding more status updates to keep our customers informed of our efforts to help our customers mitigate any effects these outages were having on their own sites.
With the resolution of the Dyn issue, we declared our services to be completely restored as of 23:37 UTC. However, we expect to make recommendations both internally and to our customers to help provide best practices to help weather these types of outages.
We want to acknowledge the pain this incident caused our customers and we are working to make some changes to our internal configurations and management of DNS to allow us to continue to provide critical services during this type of outage when possible.
We also will be updating our documentation to make sure that customers are provided the best possible information on how to set their sites up to handle outages from upstream providers. A few customers were more affected than others depending on how DNS records are configured. Refer to our documented best practices to minimize exposure to DNS failures.
For our Enterprise customers, if you have any questions about the outage or would like to discuss specific points, please file a Support ticket or contact your Acquia Account Manager.