Postmortem: INC178151 – ARU BP Clients unable to add new vendors through BH or Online Banking - Priority High
On Friday, April 5, Central 1’s bill payment clients who use ARU messaging experienced an incident between 11:40 a.m. and 2:16 pm PT (2:40 pm and 5:16 pm ET) where new billers could not be added to customer’s profiles.
The incident was resolved by restarting Central 1’s vendor validation module. During this time payments to existing biller setups were not impacted.
Point of failure: The incident was caused by a bug in the error handling of the validation routine for one specific biller. A credit union member submitted a series of invalid characters for the biller resulting in the validation library crashing because the current validation rule could not handle the error submitted to the system. Central 1 confirmed that this specific biller was originally set up on May 26, 2023, and this bug was present, however, it was undetected because the required specific conditions were not met before. To mitigate a re-occurrence the development team has addressed the bug and redeployed the biller validation to production.
Action:
PRB011244 – Root cause analysis – COMPLETED
At Central 1 we take the quality of our service delivery very seriously. We understand that when our services are not working as expected, it can have a significant impact on our customers and their businesses. We believe that the best way to address any issues that arise is to be transparent about them and work diligently to improve our processes and systems.
If you have any questions about this postmortem, please reach out to Support@central1.com.
Elizabeth Ackroyd, Manager Client Support Services (Payments) | e: eackroyd@central1.com | c: 905-301-1337