This is a summary and analysis of an issue that occurred with the delivery of an Acquia product or service. The purpose of this document is to share details about what happened and why, so there is a common understanding of what is required to prevent a future occurrence if at all possible. Any remaining issues or risks are identified, as are recommended or pending actions.
On 25 March 2020 from 18:37 to 19:58 UTC some customers attempting to view or configure environments via the Cloud UI received error messages. This also prevented customers from making environment configuration changes via the Cloud API.
Acquia identified the change responsible for the errors and began rollback procedures at 19:08 UTC. Roll back steps were completed by 19:58 UTC and normal Cloud UI / API service restored for all customers. During the time of this incident internal Acquia tools still functioned normally and Acquia Support assisted in task execution through support tickets filed through the help portal.
A change to Cloud UI / API configuration task validation steps resulted in unexpected errors. Specifically, validation checks were added for PHP settings configurable through the Cloud UI, in cases where Acquia Cloud Enterprise customers had settings configured to be outside of the value configurable within the UI an error was displayed preventing submission of tasks.
Acquia will review documentation of expected configurations / values for future changes to validation steps.
Customers should reach out to their Account team with any followup questions so that Acquia may address them on a case by case basis.