INC194858: Saturday Feb 1st e-Transfer 3.5 API service interruption

Postmortem: INC194733 - API e-Transfer 3.5 outage on Saturday, February 1 | P2

On Saturday, February 1st Central 1 experienced an e-Transfer 3.5 outage (ISO20022 send and receive) linked to an API volume threshold that was exceeded. Impact of 40 minutes for clients using ISO20022 for e-Transfers. Service was restored when ID Mapper Limits were removed as a temporary measure until new limits were established. 

Point of failure: ID Mapper API exceeded 1 million set limits. It has been sitting at 1 million for at least the last three years without previously exceeding the limit. The team is able to see that the limit has been continuously growing with normal operational growth however, they believe that the recent onboarding of a new client and month end activity caused us to reach that limit.  IDMapper API did not have monitoring on it’s service delaying the detection of the point of failure.

Actions:

PRB011471 – Root Cause Analysis

• PTASK0010287 - IDMapper API limit on AWS – Due April 2025

  • Product Support Team to work with Product team to determine and establish a new IDMapper limit. 

• PTASK0010288 - API usage for API gateway – Due April 2025

  • Building out a monitoring tool to allow the reporting and threshold monitoring on ID Mapper API usage moving forward.  This will be used to create the same monitoring capability across all API gateways.

• PTASK0010290 - NOC escalation and Re-training – Due April 2025

• PTASK0010289 - Manually Monitor ID Mapper API usage during Month end - COMPLETED

  • A manual process will be out in place to monitor the IDMapper usage by the PS Software Support Team until the until the API usage monitoring and alerting is put into place.

 We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements. 

If you have any questions about this postmortem, please contact digitalbanking_support@central1.com.

Jason Seale | Director of Client Support Services | jseale@central1.com | 778.558.5627