INC177401- Intermittent Account Summary Outage

Incident Report for Central 1

Postmortem

INC177401 - Account Summary error after Digital Banking Core Release – "Your accounts are not available at this time" – P2

On Monday, March 25th, Central 1 released digital banking core build 713 to production (CHG143843). After the 1:30 a.m. PT (4:30 a.m. ET) release members from eastern Canada started reporting Account Summary pages were not populating accounts or balances; all remaining services in online banking were not impacted (bill payments, e-Transfers, account activity, etc.). Without a known root cause, Central 1 made the decision to roll back the build at 6:13 a.m. PT (9:13 a.m. ET) resolving the incident (Status Page).

A fix was applied, and the core release was pushed live the following morning, March 26th at 1 a.m. PT (4 a.m. ET) (CHG145335) with no issues.

Point of failure: During the release, there was a missed step of applying a dependent key on the Azure servers (eastern Canada). This missing key would have only impacted members who reached the eastern Canada data center; members reaching Vancouver data center (VAHC) were not impacted (as the key was deployed there successfully). Central 1 will be enhancing our process on these types of deployments to avoid a recurrence of this type of error.

Action:

PRB011236 – Root cause analysis – COMPLETED

PTASK0010137 - Batch LDAP Modify Tool automation improvements – MD Software to complete by end of Q2 2024
PTASK0010138 - Providing VAHC and Azure Prod servers details to QA – MD Software to complete by end of Q2 2024
PTASK0010139 - Update post deployment testing to include eastern Canada servers – Release Management to complete by end of Q2 2024

At Central 1 we take the quality of our service delivery very seriously. We understand that when our services are not working as expected, it can have a significant impact on our customers and their businesses. We believe that the best way to address any issues that arise is to be transparent about them and work diligently to improve our processes and systems.

If you have any questions about this postmortem, please reach out to digitalbanking_support@central1.com.

Jason Seale, Director of Client Support Services

e: jseale@central1.com | c: 778.558.5627

Posted Apr 12, 2024 - 13:57 PDT

Resolved

This incident has been resolved. We have not detected an occurrence of the error in our logs since 6:13 am PT (9:13 am ET). We will share the details of our ongoing investigation once completed in the coming weeks.

Posted Mar 25, 2024 - 06:47 PDT

Monitoring

We have completed the rollback of all clients, and we now believe that this Incident is resolved. We will continue to monitor for the next 30-60 minutes to promote stability. Should you have any questions, please contact Digital Banking Support for assistance.

Posted Mar 25, 2024 - 06:17 PDT

Identified

The Central 1 team is actively investigating a degradation with the Account Summary page in Online banking. It has been reported that there are intermittent failures where the Account Summary page is not populating the account or balances. Subsequent functions (Bill Payments, eTransfers, Account activity) are not affected from this incident.

In order to resolve this incident, we are reverting the production release that was deployed this morning to the previous version. (CHG143843 - Digital Banking Core Release 713).

Central 1 is actively processing the rollback, and will post an update within the next 60 minutes on progress.

Posted Mar 25, 2024 - 05:42 PDT

This incident affected: Digital Banking Services and Incident Alerting.