Acquia has detected an interruption of service for some Acquia Cloud Enterprise sites
Incident Report for Acquia, Inc.
Postmortem

What happened

On Wednesday, May 23rd 2018, at 05:43 UTC Acquia was informed that a number of customers were starting to experience issues related to PHP, essentially around Cloud Hooks or Cron Jobs.

How it happened

On Tuesday, May 22nd 2018, Acquia was performing scheduled maintenance on our overall infrastructure as part of our Platform As A Service. During this process a regression was introduced which impacted the stability of your sites. A platform wide Acquia Cloud component release for PHP-CLI did not update several settings correctly causing command line PHP, Acquia Cloud Hooks and Cron tasks to use the default PHP settings instead of the Acquia settings. This issue affected the whole of Acquia Cloud and had no bearing on traffic being served correctly as this is done via PHP-FPM and not PHP-CLI.

What we did about it

Acquia Support reported this incident internally at the time and worked jointly with our Product and Operations teams to correct the issue. We also reviewed and updated our internal maintenance instructions to prevent similar problems in the future. We used the information gained during this investigation to provide further training to our release engineers. Lastly we reviewed the activities that led to this problem by adding extra pre-release validation tests and to strengthen our release process and minimise the negative impact on customer site stability in the future.

Posted Jun 04, 2018 - 19:35 UTC

Resolved
All affected sites have been restored. All services are operational at this time.
Posted May 23, 2018 - 13:52 UTC
Update
Service for some Acquia Cloud Professional, Acquia Cloud Enterprise and Acquia Cloud Site Factory sites is currently interrupted. We are working to resolve this service interruption at this time. We will provide additional updates when services are fully restored.
Posted May 23, 2018 - 11:12 UTC
Identified
Service for some Acquia Cloud Professional and Acquia Cloud Enterprise sites is currently interrupted. We are working to resolve this service interruption at this time. We will provide additional updates when services are fully restored.
Posted May 23, 2018 - 10:14 UTC
Investigating
We are currently investigating an interruption of service for some Acquia Cloud Enterprise sites. We will provide additional information as it becomes available.
Posted May 23, 2018 - 07:43 UTC
This incident affected: Cloud Platform Enterprise, Cloud Platform Professional, and Acquia Site Factory.