Acquia has detected an interruption of service for the Acquia Cloud interface and related services

Incident Report for Acquia, Inc.

Postmortem

What happened

On Monday, 2 April 2018 Acquia deployed a scheduled update to cloud.acquia.com. During this operation, the production environment’s database was dropped while initiating a deployment task; subsequently, Cloud UI, Cloud API v2, and Acquia Pipelines became unavailable.

How it happened

Due to human error, a database deployment from a lower environment to the production environment was triggered along with the code deployment.

What we did about it

When the mistake was immediately detected, the deployment task was stopped and a database restore was initiated. We updated status.acquia.com to notify customers about the outage.

Because an issue with the backup was identified, the database restore operations were re-executed. We brought the production database up to date by replaying the transactions between the backup and the last updates before the outage.

As a result of this incident we have implemented additional protection to prevent database deployments to the production environment.

Posted Apr 12, 2018 - 12:59 UTC

Resolved

The underlying cause of this service interruption has been addressed. All the Acquia Cloud interface and all related services have been restored. All services are operational at this time.

Posted Apr 02, 2018 - 19:35 UTC

Update

We continue working to restore services and will provide additional updates as they become available.

Please note that while the Cloud UI is accessible, tasks initiated through the UI are presently expected to fail and return a message indicating "This resource is temporarily unavailable."

Posted Apr 02, 2018 - 18:38 UTC

Update

We have identified the cause of the current service interruption for the Acquia Cloud interface and related services. Customers can access the Acquia Cloud interface but may be unable to take actions through the UI at this time.

We are working to restore services and will provide additional updates as they become available.

Posted Apr 02, 2018 - 17:40 UTC

Update

We have identified the cause of the current service interruption for the Acquia Cloud interface and related services. We are working to restore services and will provide additional updates as they become available.

Posted Apr 02, 2018 - 17:25 UTC

Identified

We are currently working to resolve an interruption in service for some Acquia sites and services at this time.

The following services have been restored at this time:
* Acquia Cloud interface

The following services are still impacted:
* Acquia Cloud API

Customer sites remain unaffected at this time. We will provide additional updates when services have been fully restored.

Posted Apr 02, 2018 - 17:09 UTC

Investigating

We are currently investigating an interruption of service for the Acquia Cloud interface and related services. The following sites or services are currently impacted:

* Acquia Cloud API
* Acquia Cloud interface

Customer sites are not impacted. At this time, only Acquia Cloud interface services are impacted. Currently, customers may be unable to perform administration tasks such as:

* code, database and file deployments
* on demand database backups
* domain management
* resize Acquia Cloud servers
* Cloud Drush CLI commands

We are working to restore services at this time. We will provide additional updates as they become available.

Posted Apr 02, 2018 - 16:55 UTC

This incident affected: Drupal Cloud UI.