Acquia has detected an interruption of service for the Acquia Cloud interface and related services
Incident Report for Acquia, Inc.
Postmortem

What happened

On Monday, 2 April 2018 Acquia deployed a scheduled update to cloud.acquia.com. During this operation, the production environment’s database was dropped while initiating a deployment task; subsequently, Cloud UI, Cloud API v2, and Acquia Pipelines became unavailable.

How it happened

Due to human error, a database deployment from a lower environment to the production environment was triggered along with the code deployment.

What we did about it

When the mistake was immediately detected, the deployment task was stopped and a database restore was initiated. We updated status.acquia.com to notify customers about the outage.

Because an issue with the backup was identified, the database restore operations were re-executed. We brought the production database up to date by replaying the transactions between the backup and the last updates before the outage.

As a result of this incident we have implemented additional protection to prevent database deployments to the production environment.

Posted 2 months ago. Apr 12, 2018 - 12:59 UTC

Resolved
The underlying cause of this service interruption has been addressed. All the Acquia Cloud interface and all related services have been restored. All services are operational at this time.
Posted 3 months ago. Apr 02, 2018 - 19:35 UTC
Update
We continue working to restore services and will provide additional updates as they become available.

Please note that while the Cloud UI is accessible, tasks initiated through the UI are presently expected to fail and return a message indicating "This resource is temporarily unavailable."
Posted 3 months ago. Apr 02, 2018 - 18:38 UTC
Update
We have identified the cause of the current service interruption for the Acquia Cloud interface and related services. Customers can access the Acquia Cloud interface but may be unable to take actions through the UI at this time.

We are working to restore services and will provide additional updates as they become available.
Posted 3 months ago. Apr 02, 2018 - 17:40 UTC
Update
We have identified the cause of the current service interruption for the Acquia Cloud interface and related services. We are working to restore services and will provide additional updates as they become available.
Posted 3 months ago. Apr 02, 2018 - 17:25 UTC
Identified
We are currently working to resolve an interruption in service for some Acquia sites and services at this time.

The following services have been restored at this time:
* Acquia Cloud interface

The following services are still impacted:
* Acquia Cloud API

Customer sites remain unaffected at this time. We will provide additional updates when services have been fully restored.
Posted 3 months ago. Apr 02, 2018 - 17:09 UTC
Investigating
We are currently investigating an interruption of service for the Acquia Cloud interface and related services. The following sites or services are currently impacted:

* Acquia Cloud API
* Acquia Cloud interface

Customer sites are not impacted. At this time, only Acquia Cloud interface services are impacted. Currently, customers may be unable to perform administration tasks such as:

* code, database and file deployments
* on demand database backups
* domain management
* resize Acquia Cloud servers
* Cloud Drush CLI commands

We are working to restore services at this time. We will provide additional updates as they become available.
Posted 3 months ago. Apr 02, 2018 - 16:55 UTC
This incident affected: Acquia Cloud UI.