Postmortem: INC183300 - OAUH 2SV: Error processing your request
On Thursday, July 18th, Central 1 observed a 2 Step-verification (OAuth) service degradation that started approximately at 6:50 am PT (9:50 am ET). Users who’s risk score would have prompted a step-up authentication (2SV) would have received an "Error processing your request" error. A restart of our “step-up” and OAuth services at 8:55 a.m. PT (11:55 a.m. ET) resolved the incident. Credit unions with average risk scores of 500 or > would have had approximately 5% of their members impacted, scores of 200 or > would have experienced 20% of their members impacted during the service degradation.
Point of failure:
Central 1 restarts our three 2SV services (OAuth, step-up, security-portal) several times a week. The investigating team was able to identify a service failing to restart however, the root cause for the restart failure remains unknown. The support team has been monitoring and this appears to be an isolated event possibly caused by an unexpected behavior by AWS's Web Application Firewall (WAF) resulting in corruption after the restart. There have been no recent code or configuration changes on this platform that could be tracked back to this incident. The team has turned on additional logging and will investigate further if the issue reoccurs.
Actions:
PRB011307 – Not completed yet
We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.
If you have any questions about this postmortem please contact digitalbanking_support@central1.com.
Jason Seale
Director of Client Support Services
jseale@central1.com | 778.558.5627