Incident Date: November 16, 2021
Incident Number: PR-2567
On November 16, 2021, at 6:14 AM ET, Tucows’ hosted email platform experienced service interruption, causing inbound email delays and intermittently impacting webmail in Prod A.
At 3:00 PM ET, The engineering team increased spam resources to better process the increased backlog of inbound emails and restarted AAA nodes to alleviate the high load to stabilize the hosted email environment.
The root cause is still under investigation; however, the engineering team was able to identify a malfunctioned process in the legacy Hosted Email platform that caused the Authentication nodes to high load. As a result, the Authentication node impacted, Spam filtering and Inbound mail nodes, forcing email to queue up and reducing email delivery volume.
The Tucows engineering team is to continue investigating the root cause. In addition, Tucows has increased the Hosted Email Spam filtering and Authentication nodes to address the load concerns as a preventative measure.
Tucows is committed to completing the hosted email migration into the new cloud before the end of the year to maintain a scalable and stable hosted email.
Thank you,
Tucows Engineering Team