Service outage caused by major Amazon AWS incident

Incident Report for Blue Canvas

Resolved

Blue Canvas was impacted by a major outage at AWS that affected a large number of websites. We implemented a mitigation to the impact this had on ECS and Auto-Scaling to restore service. Metadata changes were imported from Salesforce retroactively and the system is now operation normally.
Posted Nov 26, 2020 - 11:09 UTC

Update

We have implemented a mitigation to problems caused by the AWS outage. Blue Canvas is now operating normally. Please reach out if you still notice any issues or delays.
Posted Nov 25, 2020 - 23:20 UTC

Monitoring

We are in the final stages of recovery for Blue Canvas and customers should be seeing significant improvement. We expect to be fully recovered within 30 minutes.
Posted Nov 25, 2020 - 23:02 UTC

Update

We have restored read-only access to Deployment Requests. You can list and view deployments, and leave comments and reviews. Clicking the "Quick Deploy" button will currently not work. We are currently not importing changes from Salesforce. Our next step is to restore read-write access, beginning with a small subset of customers.
Posted Nov 25, 2020 - 22:31 UTC

Update

The issues with AWS are still ongoing. We are monitoring the situation and will restore service as fast as possible. Please contact Blue Canvas Support if you require urgent access.
Posted Nov 25, 2020 - 20:07 UTC

Identified

Blue Canvas is currently affected by a major AWS outage affecting multiple infrastructure services in the us-east-1 region. Our team will restore access as soon as AWS is back online. More info: https://status.aws.amazon.com/
Posted Nov 25, 2020 - 15:21 UTC

Update

We continue to investigate this issue, which may be caused by an outage in AWS US-EAST-1. Our team is currently working to restore the service.
Posted Nov 25, 2020 - 15:08 UTC

Investigating

Blue Canvas is currently experiencing a service outage. We know this is frustrating and we’re working to resolve this as soon as possible. Next update in 30 minutes.
Posted Nov 25, 2020 - 14:24 UTC
This incident affected: Blue Canvas APIs, Metadata Sync (Inbound), and Metadata Deploy (Outbound).