Postmortem: INC177486 - e-Transfer (3.5) was unavailable – P2

On Tuesday, March 26th Central 1’s 3.5 e-Transfer service (ISO20022) was unavailable between 5:07 to 5:51 p.m. PT (8:07 and 8:51 p.m. ET). Members would have seen an error ‘ try again later’ for sending and receiving 3.5 e-Transfers during the outage; unattended deposits like auto-deposit would have been received after service recovered. An error was made to a configuration of a routine change at 5:07 p.m. PT (8:07 p.m. ET) which led to the outage. The configuration was updated, and service recovered resolving the incident (44-minute outage); please note, 3.4 e-Transfer (ISO8583, EUF) was not impacted. 

Point of failure: A Central 1 staff member was completing a low-risk database (DB) update and accidently failed over the production database to a disaster recovery database.  The mistake was immediately detected, however it takes approximately 40 minutes for primary to become available again due to database synching. Firewall rules between the cloud app and the on-premise database and the inverse were not setup for a partial DR scenario contributing to the outage. 

Actions:

• PRB011235 – RCA – completed
• PTASK0010140 - Change the console color theme to differentiate between different environments in AWS – completed
• PTASK0010141 - Firewall rules for EMT APP to be configured to connect to DR Database to primary - completed 

At our company, we take the quality of our service delivery very seriously. We understand that when our services are not working as expected, it can have a significant impact on our customers and their businesses. We believe that the best way to address any issues that arise is to be transparent about them and work diligently to improve our processes and systems. 

If you have any questions about this postmortem, please reach out to Digital Banking Support (digitalbanking_support@central1.com). 

Jason Seale | Director of Client Support Services |  c 778 558 5627  e jseale@central1.com