This is a summary and analysis of an issue that occurred with the delivery of an Acquia product or service. The purpose of this document is to share details about what happened and why, so there is a common understanding of what is required to prevent a future occurrence if at all possible.
On Feb,5 2019 Acquia released a component as part of an unplanned maintenance. The release included changes to allow independent release of a file called prepend.php. During the release, environments which directly required the file site-info.php in their code base experienced PHP fatal errors on Drupal requests.
Rolling back the release of the component triggered additional failures for some customers where the reload of PHP happened after the prepend.php file was already removed.
When Acquia detected that the component release was causing some customer sites to return 500 errors, the component release was rolled back.
For some customers, the roll back caused their sites to return a different 500 error, due to the missing prepend.php. This was fixed by restarting PHP.
Acquia’s unplanned maintenance component release moved some files (/var/www/site-scripts/*). These files are included automatically where needed and the path changing was not expected to cause application impact. However, some customers on Acquia Cloud Enterprise and Acquia Cloud Site Factory had a path to site-info.php hard coded in their application, which caused a PHP fatal error as impacted Applications tried to load the file from the old path.
When the scale of the impact was realized, Acquia initiated a rollback to the preceding version. This rollback caused temporary fatal errors for any customers that saw traffic between the package downgrade and the PHP reload. Unexpectedly, for some customers, the PHP reload did not execute correctly, causing PHP fatal errors as php.ini trying to load the file that was now removed.