RESOLVED: INC200663 – Wires and Remote Deposit Capture Degradation - 7 May 2025

Incident Report for Central 1

Postmortem

Postmortem - INC200663 – PaymentStream Direct (PSD) wires outage

On Wednesday, May 7 at 8:08 a.m. PT (11:08 a.m. ET), a routine change intended to improve Kubernetes service capacity unexpectedly removed critical Wires dependencies, which restricted our PaymentStream Direct Wires (PSD Wires) application from communicating with internal dependent systems. As a result, while users could submit wires for processing, and Central 1 could receive inbound wires from external Financial Institutions, we could not process them, thus they remained in pending status until system recovery ws completed. The nature of the Incident also prevented an automated recovery, requiring extensive manual intervention to restore application functionality. Full service was restored by 11:25 a.m. PT (2:25 p.m. ET).

Point of failure: The incident was caused by human error during a production deployment. A missing configuration allowed namespace deletion, resulting in widespread service impact and a complex manual recovery process.

PRB011524 – Root cause analysis actions

Re-enable Sync Protection – COMPLETED
Review and update Helm Chart with all required secrets and configurations – COMPELTED
Review shared pipelines between Production and Disaster Recovery environments – DUE: June
Include a step in the CI/CD pipeline to regenerate MuleSoft activation tokens – DUE: June
Commit reviewers will validate changes outside of Charts – DUE: June
Implement audit log monitoring to alert when secrets or namespaces are deleted – DUE: June
Improve Disaster Recovery environments for these services – DUE: July

We recognize the disruption this caused and are committed to learning from this incident to reduce risk and improve our resiliency going forward.

If you have any questions regarding this postmortem, please contact support@central1.com

Posted May 26, 2025 - 08:48 PDT

Resolved

The wires degradation incident has been resolved, and all wires are now sending and being received through our services without delays. Wires created in PS Wires during the incident have been processed.

Additionally, there was an impact to some of our customers remote deposit capture (Deposit Anywhere) service between 8:10 to 9:10 a.m. PT (11:10 a.m. to 12:10 p.m. ET). Members would have seen an error knowing that their was a deposit issue.

Central 1 will continue to monitor our services to ensure stability and prioritize the postmortem within the next two weeks.

Central 1 - Support@central1.com - 1.888.889.7878, press 1

Posted May 07, 2025 - 11:40 PDT

Update

We continue to triage the ongoing wires disruption. We have isolated the root cause of the disruption and are working to restore full service.
Wires can still be created and reviewed in PS-Wires, however they will not be processed until this issue is resolved.

Central 1 can also receive external wires into our environment but cannot process them for our clients to accept until this matter is resolved.

An update to this incident will be provided by 11:30 a.m. PT (2:30 p.m. ET).

Central 1 - Support@central1.com - 1.888.889.7878, press 1

Posted May 07, 2025 - 10:38 PDT

Identified

Please be advised that Central 1 is experiencing wires processing degradation, preventing the completion of send/receive wires; wires can still be created and reviewed in PS-Wires. The wires API is failing due to a processing error.

We are actively investigating to restore full wire service and will provide an update by 10:30 a.m. PT (1:30 p.m. ET).

Central 1 - Support@central1.com - 1.888.889.7878, press 1

Posted May 07, 2025 - 09:12 PDT

This incident affected: Payment Services and Treasury Services.