Postmortem: INC178244 – Intermittent Login Issues on Digital Banking and INC178360 – Intermittent Login Issues on Digital Banking – P1 

On Sunday April 7th, customers accessing Central 1’s online banking for retail and small business experienced intermittent login failures between 11:20 to 11:31 p.m. PT (2:20 to 2:31 a.m. ET) (INC178244 P1). Some clients experienced continuing errors from our database until 12:40 a.m. PT (3:40 a.m. ET).  The extended impact was restricted to ‘risk challenged’ sessions for 2SV or Increased Authentication. Services fully recovered at 12:40 a.m. PT (3:40 a.m. ET) and remained stable throughout the morning peak traffic. 

On April 8th, a second event happened between 5:45 to 5:55 p.m. PT (8:45 to 8:55 p.m. ET) (INC178360 P1). During this evening period, members would have again experienced intermittent login failures on retail and small business banking.  In this second occurrence, there were no extended impacts. 

Root Cause and Remediation: Central 1 installed a vendor provided Operating System (OS) upgrade for our load balancer on Sunday, April 7th at 1 a.m. PT (4 a.m. ET). The change was classified by our vendor as routine with no production impacts expected during or after install. 

The service disruption has been attributed to an unknown bug introduced with the recent OS upgrade, where a specific condition has led to the load balancers in both datacenters losing all network port connections for several minutes.  Our vendor completed analysis of the bug, and provided an emergency patch which was completed and installed on April 8th, at 10:30 p.m. PT (1:30 a.m. ET).  The patch was completed successfully, and services have remained stabled with no further impacts. 

Central 1 is working with our vendor on improving their patch quality as well as expediting fixes when a bug is detected. 

Actions:

PRB011243 – Root cause analysis – Completed. 

At Central 1 we take the quality of our service delivery very seriously. We understand that when our services are not working as expected, it can have a significant impact on our customers and their businesses. We believe that the best way to address any issues that arise is to be transparent about them and work diligently to improve our processes and systems.

If you have any questions about this postmortem, please reach out to digitalbanking_support@central1.com. 

Jason Seale, Director of Client Support Services | e: jseale@central1.com | c: 778.558.5627