Postmortem: INC213718 - Inbound and Outbound Wires Not Processing
On Monday, December 8, beginning at approximately 8:45 a.m. PT (11:45 a.m. ET), Central 1 experienced a delay in the processing of inbound and outbound wires. Wires could continue to be created throughout the morning; however, items remained pending and did not progress through the SWIFT gateway until service was restored at 1:38 p.m. PT (4:38 p.m. ET). This resulted in an overall delay of approximately 4 hours and 53 minutes. During this period, branch staff observed growing pending queues, resulting in higher contact volumes on our wires and support desks. All items began clearing normally once the service was recovered.
Point of Failure: The processing interruption was caused by an issue in IBM’s Managed File Transfer (MFT) service, which supports moving wire files from the EPP environment to the SWIFT gateway. Earlier that morning, the IBM Payments Centre reported instability in the MFT interface. IBM’s investigation confirmed that an Inter-Process Communication (IPC) administrator, while attempting to address a customer-specific file issue, inadvertently triggered a bulk download of all customer configuration files through the MFT application. This action caused excessive Java heap consumption, resulting in performance degradation, deadlocks across multiple clients, and partially responsive MFT nodes.
Although backend processing components were still functioning, the degraded state of MFT prevented wire files from progressing, resulting in the growing pending volumes observed by credit unions. The absence of an automated alert tied to wire-pending volumes delayed escalation, as all external monitoring systems appeared healthy until the service impact was manually identified.
Following confirmation of a system-wide effect, IBM performed an MFT restart at approximately 12:00 p.m. ET, which restored file movement by 12:11 p.m. ET. A small number of clients saw minor performance lag until the remaining deadlocks were fully cleared from the database.
Corrective Actions:
Central 1 and IBM have initiated an investigation into an ongoing problem under PRB011641. Immediate remediation included reducing heap thresholds to trigger earlier warning signals, instituting new file-count monitoring rules to detect abnormal UI activity, and revalidating documentation governing administrative actions within the MFT interface. IBM and the vendor are also progressing with a product upgrade to MFT Version 7.9.1, scheduled for completion in February 2026, which is expected to reduce the likelihood of UI deadlocks significantly. Additional work is underway to validate configuration download limits within the product and to explore role-restricted administrative access to prevent similar incidents.
As part of our internal improvement stream, Central 1 is reviewing automated threshold-based alerting for pending wire volumes to provide earlier visibility and escalation, ensuring vendor notifications and operational checks consistently surface anomalies before they impact clients.
We apologize for the delay and appreciate the patience while Central 1 works with our vendors to improve our systems high-availability.
Central 1 - Support@central1.com - 1.888.889.7878