INC194858: Saturday Feb 1st e-Transfer 3.5 API service interruption

Incident Report for Central 1

Postmortem

Postmortem: INC194733 - API e-Transfer 3.5 outage on Saturday, February 1 | P2

On Saturday, February 1st Central 1 experienced an e-Transfer 3.5 outage (ISO20022 send and receive) linked to an API volume threshold that was exceeded. Impact of 40 minutes for clients using ISO20022 for e-Transfers. Service was restored when ID Mapper Limits were removed as a temporary measure until new limits were established. 

Point of failure: ID Mapper API exceeded 1 million set limits. It has been sitting at 1 million for at least the last three years without previously exceeding the limit. The team is able to see that the limit has been continuously growing with normal operational growth however, they believe that the recent onboarding of a new client and month end activity caused us to reach that limit.  IDMapper API did not have monitoring on it’s service delaying the detection of the point of failure.

Actions:

PRB011471 – Root Cause Analysis

  • PTASK0010287 - IDMapper API limit on AWS – Due April 2025

    • Product Support Team to work with Product team to determine and establish a new IDMapper limit. 
  • PTASK0010288 - API usage for API gateway – Due April 2025

    • Building out a monitoring tool to allow the reporting and threshold monitoring on ID Mapper API usage moving forward.  This will be used to create the same monitoring capability across all API gateways.
  • PTASK0010290 - NOC escalation and Re-training – Due April 2025

  • PTASK0010289 - Manually Monitor ID Mapper API usage during Month end - COMPLETED

    • A manual process will be out in place to monitor the IDMapper usage by the PS Software Support Team until the until the API usage monitoring and alerting is put into place.

 We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements. 

If you have any questions about this postmortem, please contact digitalbanking_support@central1.com.

Jason Seale | Director of Client Support Services | jseale@central1.com | 778.558.5627

Posted Mar 09, 2025 - 12:12 PDT

Resolved

Please be informed that Central 1 experienced a service disruption with our e-Transfer 3.5 API services on Saturday, February 1, from 11:23 a.m. to 12:03 p.m. PT (2:23 to 4:03 p.m. ET). The service was restarted to clear sessions which resolved the issue. Please note, 3.4 API e-Transfer and all other e-Transfers were not impacted.

Central 1 will provide a postmortem within the next two weeks.

Client Support Services
T 888 889 7878 Option 2 - DigitalBanking_Support@central1.com
Posted Feb 03, 2025 - 09:03 PST
This incident affected: Payment Services.