Postmortem: INC183294 - Lynx Incident — Swift (Wires) Outage | P2
Payments Canada notified Central 1 that the Wires Swift traffic was partially degraded on Thursday, July 18, 2024, between 2:57 a.m. PT and 7 a.m. PT (5:57 a.m. to 10 a.m. ET). Customers could still submit wires, however, processing and delivery of outgoing and incoming wires via Swift were impacted. Wires being sent and received to other Central 1 customers (B2B) were not impacted. Processing was restored at 7 a.m. PT (10 a.m. ET) when normal operations of wires Swift services resumed.
Point of failure: The incident was caused by a software defect in the SwiftNet Flow Module (SFM) which orchestrates copies and workflows across services, such as in-flow translation.
The incident’s root cause is linked to code introduced in June 2020, where this sequence number was wrongly defined as a 32-bit integer in one of the internal communication components. This limit was reached on July 18, 2024, after four years of continuous processing. As a result, the SFM application stopped working as it was unable to reconcile Event ID numbers across various processing threads (as the 64-bit event IDs continued to increase while the corresponding 32-bit were being truncated).
The problem was fixed by resetting the value of the SFM Event ID number for all new incoming flows and updating the references for all other traffic pending processing. Once these updates were performed, traffic resumed normally.
Actions:
To avoid any reoccurrence, a SWIFT software release is required by Payments Canada. The initial reset conducted on July 18, 2024, will ensure continuity for the next 18 months at current traffic levels. The permanent solution will be implemented within this 18-month timeframe.
We sincerely apologize for the disruption to our Wire services. Please know that we are fully committed to learning from this incident and are working tirelessly to enhance our service reliability. Thank you for your understanding and continued trust in us.
If you have any questions about this postmortem, please contact Support@central1.com.
Elizabeth (Liz) Ackroyd
Manager, Payments
Client Support Services Central
Certified Lean Greenbelt Champion
2810 Matheson Blvd East, Mississauga, ON, Canada L4W 4X7
T 1 800 661 6813 ext. 8583 C 905 301 1337 Support 888 889 7878