Notice - Unscheduled Maintenance for your Acquia Search Services
Incident Report for Acquia, Inc.
Postmortem

Purpose of This Report

This is a summary and analysis of Acquia’s response to the log4j vulnerability (CVE-2021-44228). Any remaining issues or risks are identified, as are recommended or pending actions.

Event Summary

On December 10th, 2021, Acquia was made aware of a security vulnerability in the Apache log4j logging utility.  Acquia immediately investigated and determined that Acquia Search with Solr 7 was the only customer-accessible Acquia product impacted by the log4j vulnerability, CVE-2021-44228. 

Due to the nature of this security vulnerability, Acquia chose to deactivate all possible susceptible search services while investigation and remediation took place in order to protect all Acquia customers and their data. This action was taken with knowledge that it could adversely impact applications and occurred with approval from all executive stakeholders given the potential risk of remote code execution.  In most cases, the impact to applications was that search requests resulted in errors being returned.  In some cases, applications may not have been able to ‘fail gracefully’ when search requests resulted in errors or where connections to search services resulted in requests being unable to timeout.  Acquia seeks to maintain a stable and secure platform.  In this case, the total security of all customer applications and customer data was required to take precedent over all other considerations.

Testing by Acquia security personnel found that in many cases Drupal and/or its associated modules prevented exploitation. Acquia ensured that the vulnerability was mitigated fully before reactivating the affected search services. 

Further patching of log4j to version 2.16.0 was conducted 16 December.  This maintenance was also conducted on an urgent basis; however, review of data during this second round of actions for all search infrastructure indicated that no interruption of service took place and impact was minimal across all systems.

No Acquia hosting platform (Acquia Cloud Enterprise both Classic and Next, Acquia Cloud Site Factory, and Acquia Cloud Professional) makes use of Java and thus does not directly make any use of log4j.

Acquia has, further, consulted with all 3rd party vendors to confirm that all systems for those vendors have also been mitigated for CVE-2021-44228.

As a standard practice, Acquia recommends that customers design their applications and all associated forms that accept input from users to evaluate and sanitize all input. Acquia evaluates and implements mitigations to known vulnerabilities according to our security policies and reminds all customers that security is a shared responsibility between the platform and applications.

Acquia Actions

All times UTC

  • 2021-12-10

    • 05:00 - Acquia received a first notice of CVE-2021-44228 and security personnel began investigating.
    • 08:50 Acquia Search with Solr 7 was initially identified as being possibly vulnerable and requiring mitigations to be put in place.
    • 15:00 - Initial communications regarding unscheduled maintenance were sent to customers.
    • 15:30 - Acquia Search platform was taken offline to begin immediate implementation of mitigations for CVE-2021-44228
    • 17:45 - An updated communication was sent to customers regarding the maintenance in progress.
    • 19:00 - Patching was completed for Acquia Cloud Search and Search services were re-enabled.
  • 2021-12-11

  • 2021-12-15 

    • 17:00 - Updated public guidance regarding log4j indicates there was an additional low risk vulnerability.  At this time emergency patching was not anticipated.  This information was also provided via the above Knowledge Base article
    • 21:00 -  Further public guidance indicates that the additional identified vulnerability is of higher severity than initially communicated.  Again, the linked knowledge base article was updated to indicate the maintenance would be conducted on an emergency basis.
  • 2021-12-16 

    • 10:30 - Release of 2.16.0 to Acquia Search with Solr 7 was initiated.
    • 13:30 - A further round of communications with all customers whose search application could have been impacted by maintenance actions was sent.
    • 14:00 - Release to Search with Solr 7 was completed.

Corrective Actions

  1. Acquia will further patch log4j to version 2.17.1 and will be subject to Acquia's standard patching procedure.
Posted Dec 29, 2021 - 21:37 UTC

Resolved
All services are operational at this time. A postmortem document with more information regarding the vulnerability and Acquia actions to remediate it will be published once available.
Posted Dec 10, 2021 - 22:16 UTC
Monitoring
The unscheduled maintenance has been completed and the zero day vulnerability has been mitigated. As a result, the Acquia Search Service has been restored. Customers should now be able to use Search v3 within their applications. We continue to monitor the services at this time.
Posted Dec 10, 2021 - 19:21 UTC
Update
Acquia is performing unscheduled security maintenance to the Acquia Search with Solr 7 to remediate a widespread zero day vulnerability. In order to conduct this maintenance and prevent any possible exploitation of Acquia Search with Solr 7 access has been disabled. This will impact applications that are reliant on Solr. At present Acquia has not found any cases where this vulnerability has been exploited on the Acquia Platform.

Maintenance actions will begin as follows:
Dublin (eu-west-1) on - 10 December 2021, 2:00pm GMT
Frankfurt (eu-central-1) on - 10 December 2021, 3:00pm CET
São Paulo (sa-east-1) on - 10 December 2021, 11:00am BRT
Singapore (ap-southeast-1) on - 10 December 2021, 10:00pm SGT
Sydney (ap-southeast-2) on - 11 December 2021, 1:00am AEDT
Tokyo (ap-northeast-1) on - 10 December 2021, 11:00pm JST
US East (us-east-1) on - 10 December 2021, 9:00am EST
US West (us-west-2) on - 10 December 2021, 6:00am PST
Canada Central (ca-central-1) on - 10 December 2021, 9:00am EST
UTC on - 10 December 2021, 2:00pm UTC
Posted Dec 10, 2021 - 17:58 UTC
Identified
Acquia is performing unscheduled security maintenance to the Acquia Search Services in order to patch a current widespread security vulnerability affecting applications. The potentially affected components have been taken off line until the patching is complete in order to ensure that applications remain secure.

Maintenance actions will begin as follows:
Dublin (eu-west-1) on - 10 December 2021, 2:00pm GMT
Frankfurt (eu-central-1) on - 10 December 2021, 3:00pm CET
São Paulo (sa-east-1) on - 10 December 2021, 11:00am BRT
Singapore (ap-southeast-1) on - 10 December 2021, 10:00pm SGT
Sydney (ap-southeast-2) on - 11 December 2021, 1:00am AEDT
Tokyo (ap-northeast-1) on - 10 December 2021, 11:00pm JST
US East (us-east-1) on - 10 December 2021, 9:00am EST
US West (us-west-2) on - 10 December 2021, 6:00am PST
Canada Central (ca-central-1) on - 10 December 2021, 9:00am EST
UTC on - 10 December 2021, 2:00pm UTC
Posted Dec 10, 2021 - 15:02 UTC
This incident affected: Acquia Search.