Postmortem: INC194733 - API e-Transfer 3.5 outage on Saturday, February 1 | P2
On Saturday, February 1st Central 1 experienced an e-Transfer 3.5 outage (ISO20022 send and receive) linked to an API volume threshold that was exceeded. Impact of 40 minutes for clients using ISO20022 for e-Transfers. Service was restored when ID Mapper Limits were removed as a temporary measure until new limits were established.
Point of failure: ID Mapper API exceeded 1 million set limits. It has been sitting at 1 million for at least the last three years without previously exceeding the limit. The team is able to see that the limit has been continuously growing with normal operational growth however, they believe that the recent onboarding of a new client and month end activity caused us to reach that limit. IDMapper API did not have monitoring on it’s service delaying the detection of the point of failure.
Actions:
PRB011471 – Root Cause Analysis
PTASK0010287 - IDMapper API limit on AWS – Due April 2025
PTASK0010288 - API usage for API gateway – Due April 2025
PTASK0010290 - NOC escalation and Re-training – Due April 2025
PTASK0010289 - Manually Monitor ID Mapper API usage during Month end - COMPLETED
We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.
If you have any questions about this postmortem, please contact digitalbanking_support@central1.com.
Jason Seale | Director of Client Support Services | jseale@central1.com | 778.558.5627