What happened
On August 23, 2018 Acquia released a component update for an internal platform service on the web server tier. Acquia runs a specific daemon on all web servers for clients who have a high site count which, on incoming requests, determines if there is enough memory available on the system to serve the incoming request. If PHP is not already running it will first request information from this daemon to determine if it's ok to launch a new process. As part of our ongoing maintenance we deployed a change to this daemon. In our testing and in our pre production deployments we did not observe any problems with this change and our expectation was that there would not be any customer impact.
Hours after the deployment was complete, Acquia started receiving automated and customer submitted alerts that some sites were experiencing problems. Initial investigation narrowed down the likely culprits to the daemons that are responsible for launching new processes in a specific PHP configuration.
What we did about it
None of the initial attempts to restart the daemon had the desired effect so a decision was made to turn the specific functionality related to the offending PHP configuration off. While this change had a positive impact for some sites it caused an even bigger problem for some servers running a high number of sites where the PHP processes did not have enough memory to complete requests as expected.
At this point it was decided that the release which introduced the change to this daemon needed to be rolled back. During the rollback process a subset of customers experienced problems running their scheduled cron jobs. After the roll back was completed sites availability was restored.
Identified Root Cause
The resource control daemon stopped handling connections from customer applications utilizing a specific PHP configuration. While other failure modes of the resource control daemon are handled automatically, this case was not and caused PHP processes to hang or deny requests that normally would be handled.