OpenSRS Status - Cluster B - Mailstore 21, 23, 25, 27, 29 Login issues

Cluster B - Mailstore 21, 23, 25, 27, 29 Login issues

Incident Report for OpenSRS

Postmortem

Incident Date: October 1, 2021
Incident Number: PR-2412

On October 1, 2021, at 2:34 PM ET, Tucows’ hosted email platform experienced service interruption impacting IMAP, POP and Webmail in Prod B and MAC B for Exact hosting.

The service interruption was due to executing of an approved low-risk change causing a split-brain issue on one of the load balancers.

At 3:41 PM ET, The Engineering team recovered the services by restarting the impacted load balancers to stabilize the email environment.

Tucows is to further enhance the triage and troubleshooting documentation to resolve the issues in a timely manner

Tucows is to further revise and improve the change management process for better visibility and faster recovery.

Thank you,

Tucows Engineering Team

Posted Oct 13, 2021 - 14:58 UTC

Resolved

All the services have recovered successfully and customers will be able to log in to Webmail.

Incident Start Time: 10-01-2021 18:34:00
Incident Start Time:10-01-2021 19:41:00
Total Duration: 1 hour 7 mins

Posted Oct 01, 2021 - 20:09 UTC

Identified

Our Engineering team is troubleshooting the failover issue and continues to restore the failed services.

Posted Oct 01, 2021 - 19:28 UTC

Investigating

Users on mailstore 21, 23, 25, 27, 29 are having issues with login. Our Engineering team has been engaged and working on it.

Posted Oct 01, 2021 - 19:03 UTC

This incident affected: Hosted Email (Cluster B, Webmail).