Postmortem: Incident – INC160638 - Incoming wires not available to process – Priority High

Overview

On Sunday, June 4, 2023, at 10 p.m. PT, EPP started sending Zenoss EPP MuleSoft connection alerts. Central 1 opened a critical priority ticket with Fiserv (CS08397917) at 12:45 a.m. PT which was acknowledged by Fiserv at 12:54 a.m. PT.. EPP was not available from 4:30 a.m. PT (7:30 a.m. ET) to 8 a.m. PT (11:30 a.m. ET) or processing incoming wires.

Fiserv did not make any changes over the weekend. To fix the incident they rebooted their Firewall after which EPP became available.

The incident was caused by the Fiserv Palo Alto Firewall ethernet connections being in a down state thereby disconnecting Central 1’s connection to the Fiserv-hosted EPP environment.

Incident Ticket:

Opened: 06-05-2023 12:13 a.m. PT (3.13 a.m. ET)

Resolved: 06-05-2023 9:57 a.m. PT (12.57 p.m. ET)

Fiserv Actions

There was no indication from a platform perspective that the Palo Alto VMs went offline.  A ticket has been opened with Palo Alto to analyze the OS logs and provide insights into if there was any service disruption caused by OS on both devices.

We recognize that there is always room for improvement, and we are fully committed to continuously evaluating and improving our processes and vendor stability to deliver the highest quality services to our customers. We thank all our customers for their patience and understanding as we work to improve our service delivery.

If you have any questions about this postmortem, please reach out to me.

 Liz Ackroyd

Manager, Client Support Services, Payments

Eackroyd@central1.com| 905.301.1337