Postmortem: MemberDirect Business (MDB) Intermittent Login Failure | INC152535 P2
Summary
On Monday, January 23 starting at 4:46 p.m. PT (7:46 p.m. ET) MemberDirect Business (MDB) clients started to sporadically receive a “Page cannot be reached” error when trying to access MDB online banking. The error was happening approximately 50 per cent of the time. In some cases, a retry to login found success. The error was detected by Central 1’s 24x7 site monitoring, however on-call resources were not engaged as the site did not register as offline. Central 1 started to receive calls from clients to report the subject on Tuesday, January 24 at approximately 9:30 a.m. PT (12:30 p.m. ET). The incident was resolved on Tuesday, January 24 at 10:40 a.m. PT (1:40 p.m. ET). Our Status Page notice was closed on Tuesday, January 24 at 3:12 p.m. PT (6:12 p.m. ET) after confirming system stability.
Postmortem
Central 1 has determined that the incident was due to an IP error on Central 1’s web traffic load balancer. The incident affected production for a total of 17 hours and 44 minutes.
An IP error was discovered on Central 1’s web traffic load balancer and a workaround to force a stable IP was implemented at 10:30 a.m. PT (1:30 p.m. ET) resolving the incident. The incident affected production for a total of 17 hours and 44 minutes.
Known point of failure: Central 1’s Global Server Load Balancing (GSLB) started returning 2 IP addresses for MDB (business.memberdirect.net) at 4:46 p.m. PT (7:46 p.m. ET) on January 23, 2023. The GSLB was returning IP addresses from both on-prem data centers (Toronto (TOHC) and Vancouver (VAHC)). Members presented with the TOHC (east coast clients primarily, due to geolocation) IP address would see the error. To resolve the incident, the VAHC application servers were added to the TOHC resource pool.
The Central 1 Network team opened a ticket with our vendor (F5, our GSLB vendor) to find root cause. F5 recommended a change to memory provisioning along with some patching and maintenance to our load balancer that was experiencing the problem. The memory sizing changes were applied on January 29th (CHG132115) during the Central 1 maintenance window.
Actions
PRB011045 – F5 Services Review
Due Date: End of April
RITM327059 - Site 24x7 alert improvements during flapping events
Due Date: End of March
Central 1 is working on our technical debt and service improvements within our Operational Excellence program. We are seeing progress already in this initiative and hope to mitigate the occurrences of these member impacting incidents as we move forward. We apologize for any inconvenience to your business members and are working hard to earn trust in our service stability.
Central 1 is completing a detailed review of our load balancer with F5 to ensure all patching and synchronization has been completed and is up to date.
Please reach out to me directly if you have questions about this incident report. I am also available to answer questions about Central 1's program to improve service availability and mitigate outages.
Jason Seale
Director, Client Support Services
Central 1 Client Support Services
c 778 558 5627 | jseale@central1.com