Postmortem: INC189678 – Digital Banking Outage (25 minutes) – P1
On Friday November 1st Central 1 experienced a service degradation on the Forge and MemberDirect platform, impacting login and some services for both Desktop and mobile application users between 8:25 to 8:50 a.m. PT (11:25 to 11:50 a.m. ET). The online banking degradation lasted approximately 25 minutes when services fully recovered on their own.
Point of Failure: Investigation into root cause found that a combination of higher-than-normal Interac transaction processing and the vendor Interac system responding slowly resulted in exhausting our proxy server. On Friday, the load and slowness of the vendor system held up resources on the proxy server using up available threads in the Vancouver datacentre. To permanently fix this issue Central 1 will be adding another proxy server to the Vancouver datacenter, this will ensure we have required capacity to deal with increased transactions and vendor latency during month end.
Actions:
PRB011434 – Root Cause Analysis
We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.
If you have any questions about this postmortem, please contact digitalbanking_support@central1.com.
Jason Seale
Director of Client Support Services
jseale@central1.com | 778.558.5627