Acquia has detected an interruption in our ability to launch new sites for Acquia Cloud Free, Acquia Cloud Professional, and Acquia Cloud Enterprise
Incident Report for Acquia, Inc.
Postmortem

Purpose of This Report

This is a summary and analysis of an issue that occurred with the delivery of an Acquia product or service. The purpose of this document is to share details about what happened and why, so there is a common understanding of what is required to prevent a future occurrence if at all possible. Any remaining issues or risks are identified, as are recommended or pending actions.

What happened

On August 31, 2018 Acquia detected a problem with launching or relauching server instances in the Acquia Cloud. This caused Acquia to halt any provisioning or configuration change operations in Acquia Cloud Enterprise, as well as turning off any resizing or purchasing operations for Acquia Cloud Professional. After identifying downtime in a third party service that is called for all Operating System launches, Acquia disabled this service across the complete Acquia Platform and then re-enabled all provisioning, launch, and resizing operations.

Identified Root Cause

The tools and processes that manage the provisioning and launching of server instances in the Acquia Cloud have a dependency on a service to verify the operating system patch level when server instances are created. This service experienced downtime and caused the Acquia tools to fail.

Next Steps

  1. Acquia will change tooling and processes to be fault tolerant to external services
  2. Acquia will improve internal detector tooling to alert earlier for server instance launch failures
Posted 19 days ago. Sep 05, 2018 - 16:32 UTC

Resolved
All affected Acquia services have been restored. All services are operational at this time.
Posted 24 days ago. Aug 31, 2018 - 22:25 UTC
Monitoring
At this time, site launch services for the Acquia Platform have been restored. Customers with Acquia Cloud Professional sites can upsize their existing sites as needed.

We are currently monitoring the ongoing service interruption issues and will provide additional updates when services are fully restored.
Posted 24 days ago. Aug 31, 2018 - 19:34 UTC
Identified
Site launch services for on the Acquia Platform are currently interrupted. We are working to resolve this service interruption at this time. We will provide additional updates when services are fully restored.

Additionally, customers with Acquia Cloud Professional sites cannot currently upsize their existing sites. No active sites are currently impacted.

We will provide additional information as it becomes available.
Posted 24 days ago. Aug 31, 2018 - 17:53 UTC
Investigating
We are currently investigating an interruption of service for launching new sites in Acquia Cloud Free, Acquia Cloud Professional, and Acquia Cloud Enterprise. Customers with sites on Acquia Cloud Free, Acquia Cloud Professional, and Acquia Cloud Enterprise are unable to launch new sites at this time. Additionally, customers with Acquia Cloud Professional sites cannot currently upsize their existing sites.

No active sites are currently impacted. We will provide additional information as it becomes available.
Posted 24 days ago. Aug 31, 2018 - 17:10 UTC
This incident affected: Acquia Cloud Enterprise, Acquia Cloud Professional, Acquia Cloud Site Factory, and Acquia Cloud Free.