.png)
1 hour ago
1
PagerDuty, the incident management platform used by thousands of organisations to alert them to problems on their systems, suffered a major outage itself on 28th August 2025. In a comprehensive outage report, the company detailed the scope of the problem, the customer impact, and how it is working to prevent a recurrence.
The incident disrupted or delayed the processing of incoming events to customers in PagerDuty's US service region. Significant service degradations affected PagerDuty for more than nine hours. At its peak, approximately 95% of events were rejected over a 38-minute period, and 18% of create requests generated errors for 130 minutes.
According to the outage report, the cause was a bug in a new feature being rolled out to improve the auditing and logging of API and key usage. As the incremental rollout progressed, usage on PagerDuty's Kafka clusters erroneously grew past the system's capacity.
Due to a logical error in the aforementioned feature, a new Kafka producer was instantiated for every API request, rather than using a single Kafka producer to produce messages.
The report explains that PagerDuty's interpretation of how to use the pekko-connectors-kafka Scala library caused this coding error. The report details the scope of the extra load: "Kafka ended up tracking nearly 4.2 million extra producers per hour at peak. This is 84 times higher than our typical number of new producers." It goes on to explain how Kafka started thrashing and then exhausting the JVM heap available to it, causing a cascading failure of the cluster.
Because so many of our systems depend on Kafka, this slowdown spread, and services eventually lost the ability to interact with Kafka entirely
This caused a chain reaction in other services, which depend on Kafka for communication, as they could not connect to the Kafka cluster. This increased the impact of the outage and the time it took to recover the situation. The company acknowledged that "cascading failures are inherently hard to predict, and a small issue in one service can ripple into others in ways not obvious from system diagrams."
Adding to the irony of an incident affecting one of the leading incident management platforms, there were delays in external communication as updates drafted by PagerDuty staff did not appear on the public status page. Customer confusion was increased further, as this "meta-failure" meant that they couldn't get updates about the outage as it went on.
PagerDuty is not the only incident management platform to suffer an extended outage in recent times, with Opsgenie customers experiencing a 14 day outage back in 2022.
Reaction from the community showed how critical a reliable incident management system can be to modern organisations, with one poster to Reddit explaining the stress caused by the lack of visibility of their systems during the outage: "today was supposed to be a big day at work. instead i spent it getting yelled at by customers because pagerduty crapped out... you ever been oncall and feel like you're just blind?" User Vimda followed up by suggesting adding redundancy to all systems: "Always have a backup Alerting system, even if it's manual triage". User Twirrim doubled down on this, reflecting thoughtfully that monitoring tools themselves need monitoring, stating that "single points of failure are the biggest enemies to reliability. Sometimes they're unavoidable (such as being prohibitively expensive), so you have to account for 'what if it goes wrong.'"
Alongside the detailed timeline of the outage and what caused it, PagerDuty also listed some future improvements and commitments to avoid any repeat scenario, including expanding their own monitoring, especially at the JVM and Kafka level, and implementing stricter change management guardrails so that engineers can still work quickly but with added safety.
The community reaction validates the need for all organisations to make sure their own systems and processes are resilient, by consciously ensuring there is redundancy and backup plans in the case of third-party outages. PagerDuty's own outage report and immediate plans for future improvements reflect a strong and psychologically safe culture.
PagerDuty’s culture of continuous learning means we emerge from incidents like this stronger – in both our technology and our team.
