Incident Date: July 8, 2021
Incident Number: PR-2142

On July 8, 2021 at 11:57 AM ET, Tucows’ hosted email platform experienced service interruption causing inbound and outbound email delays for Prod A. Tucows’ Engineering team was engaged to investigate the issue.

The service interruption was due to execution of a planned maintenance that caused a rspam email processing loop.

At 12:23 PM ET, the Engineering team reverted the change and started monitoring the issue. Since no further errors were observed, the incident was marked resolved. 

At 2:04 PM ET, we noticed further impact to outbound emails in Prod A. At 3:00 PM ET, the engineering team added additional resources on rspam to handle the high load on the system. 

On July 9, 2021 at 1:43 PM ET, The engineering team was able to successfully revert the changes to the legacy system and restore the services. 

Tucows is committed to implementing preventive measures by further deploying new configuration to ensure better email processing, enhancing monitoring and change management processes for better visibility, QA validation and faster recovery.

Thank you,

Tucows Engineering Team