Central 1 Vendor’s Postmortem:

Incident Outage: June 8th from 5:48 a.m. to 8:00 a.m. PT (6:48 a.m. to 11 a.m. ET) – 2 hours and 12 minute service degradation for processing of SMS traffic

Summary: Telesign system global SMS traffic degradation

Root Cause: Following the successful staging testing of the routine set of updates to our global SenderID rules database, Telesign started rolling out the changes to the production environment. Shortly after the initiation of the rollout (June 8, 2023, 12:48 UTC), Telesign’s NOC Team observed an increase in operational load on the SMS platform, due to which our incoming SMS traffic (SMS Verify and SMS API products) started building up processing queues in our data centers. Initially, this has caused all of the incoming SMS traffic to be accepted by Telesign but not processed as intended and sent to the Customers’ end users.

Telesign immediately started the work on mitigating the issue and after a number of different attempts, at June 8, 2023, 15:10 UTC, final rollback of the update was completed by Telesign’s Technical Operations Team, in an effort to clear the queue and return to normal traffic processing. From June 8, 2023, 16:43 UTC all remediation activities were completed, and SMS traffic processing resumed in full capacity.

Preventive Measures: To minimize the risk of, and/or prevent this issue from recurring in the future, Telesign’s Team has taken the following actions:

Completed actions:

• Telesign updated the procedures and relevant documentation to restrict the scope of configuration changes that can be applied in a single production rollout.
• Telesign has performed scaling-up of its hardware resources, enabling the platform to manage faster recovery during unexpected events.

Ongoing actions:

• Telesign will introduce additional verification checks in the staging environment to validate configuration changes of this type more confidently.
• Telesign will expand its simulation of global load test capabilities in the staging environment to determine the configuration change will not affect the infrastructure during the planned production rollout.
• Telesign will update its production rollout procedure around failed rollout scenarios to further reduce the appropriate action’s required time (rollbacks and remediations).

 We apologize for the inconvenience this may have caused you. Should you have any questions, please don’t hesitate to contact us.

 -----------------

Digital Banking Support

Central 1 Client Support Services

T 888 889 7878,   Option 2 | DigitalBanking_Support@central1.com