The Ops Community ⚙️: Mandi Walls

Resources for Tackling Alert Fatigue with SLOs, Automation, and Machine Learning

Mandi Walls — Thu, 11 Sep 2025 21:03:45 +0000

These are the resources listed in my talk "Tackling Alert Fatigue with SLOs, Automation, and Machine Learning".

Have any other tips? Let us know in the comments!

Understanding Alert Fatigue and How to Prevent It https://www.pagerduty.com/resources/learn/alert-fatigue/
8 Ways to Reduce Alert Fatigue https://www.pagerduty.com/blog/reduce-alert-fatigue/
What’s the Difference Between SLAs, SLOs, and SLIs? https://www.pagerduty.com/resources/learn/what-is-slo-sla-sli/
What is AIOps? https://www.pagerduty.com/resources/learn/what-is-aiops/
Automated Remediation https://autoremediation.pagerduty.com/
Google SRE Workbook https://sre.google/workbook/implementing-slos/
Nobl9 https://www.nobl9.com/

It's almost time for re:Invent

Mandi Walls — Tue, 26 Nov 2024 16:20:04 +0000

You wanted to have a nice relaxing Thanksgiving, sit at home or go see family. Have some good food. Watch a big ol' parade and some sportsball on TV.

Looming on the horizon, though. What is that? It's not just the Sunday Scaries.

It's re:Invent. The big AWS tech show in Las Vegas, sitting on the shoulder on the holiday.

I will be there, in the PagerDuty booth. Stop by and say hi. ;)

Here's some top things that come up for folks who don't go to Vegas often or ever.

It's DRY

I don't mean "don't repeat yourself", I mean the humidity. It's desert. If you're coming in from a more temperate region, you'll feel it.

It will feel like the air is sucking the moisture out of your body. And your eyeballs.

Prepare yourself:

Moisturizer for your skin.
Chapstick or your favorite lip balm.
Eye drops. I'm not kidding. Sucking the moisture from your eyes.
Nasal spray or gel. I like AYR Saline Nasal Gel, which is available at most drug stores. Helps if you're prone to nosebleeds in dry weather.
Water bottle. Or you could hope to score one in the swag. :D

It's Pricey

It's a tourist trap in the desert. Of course everything is going to be expensive.

It's Packed

If you've read about any nice restaurants or experiences in Las Vegas that you'd like to check out, the time to reserve that was in the spring.

Many of the restaurants in the Venetian will be bought out by sponsors to use for meetings and shmoozing. The other popular eateries on the strip will also be booked out for meetings.

re:Invent won't even be the only show going on. This year the other big show will be Marijuana Business Daily - MJBizCon 2024 at the Convention Center, bringing another 30k people.

It's Far

Nothing is close together. The Venetian itself is gigantic.

If you're staying at one of the sleeping hotels, give yourself at least an hour to get to your sessions.

What are your survival tips?

Let me know in the comments!

Resources for Best Practices for On Call Teams

Mandi Walls — Wed, 17 Apr 2024 14:11:32 +0000

Hi! This page is a set of resources and reference for folks who have attended our "Best Practices for On Call Teams" talk, but the list below will be helpful for anyone looking to improve their on call or build a new on call practice for their technical team.

If you haven't seen the talk, there is a version posted here.

PagerDuty Resources

Going On Call Ops Guide
For step-by-step instructions for setting up your team in PagerDuty, see this On-Call Rotations and Schedules resources page
How to Get Notified Before You Go On-Call in PagerDuty
Sign up for our On Call Guide e-book
Keep an eye on our events page https://www.pagerduty.com/events/ for meetups, webinars, PagerDuty Connects, and other opportunities
For in-depth training check out PagerDuty University: https://www.pagerduty.com/university/
Join the PagerDuty Community at https://forums.pagerduty.com and the Rundeck EMEA meetup at https://www.meetup.com/rundeck-europe/

Industry Resources

The informal survey of on call compensation from The Pragmatic Engineer.
Increment, a magazine published by Stripe published an issue about on-call as their very first issue https://increment.com/on-call/
Alice Goldfuss’s open source on-call handbook: https://github.com/alicegoldfuss/oncall-handbook
New Relic shares some of their best practices for on-call, as well as their incident response workflows https://blog.newrelic.com/engineering/on-call-and-incident-response-new-relic-best-practices/
In this classic session from the Velocity conference, Etsy’s team talks about how they worked to quantify their on-call. Mean Time to Sleep https://www.youtube.com/watch?v=FLqucVb_et0&feature=youtu.be&ab_channel=LaurieDenness

Psychological Safety

Fostering Psychological Safety guide by Google
Our podcast episode on psychological safety with Tom Geraghty.

More resources at https://goingoncall.pagerduty.com/resources

Resources for Plan for Unplanned Work: Game Days and Chaos Engineering

Mandi Walls — Wed, 17 Apr 2024 14:10:37 +0000

Helpful resources mentioned in my talk:

Industry Resources

PagerDuty Resources

https://response.pagerduty.com
PagerDuty Live: Failure Fridays with Rich Lafferty and Stevenson Jean-Pierre
On the blog: 10 Years of Failure Fridays at PagerDuty

Resources: Monitoring and Observability Panel Discussion

Mandi Walls — Wed, 17 Apr 2024 14:06:58 +0000

These resources were referenced during our panel discussion held in November, 2023. The recording is available on our YouTube channel.

DORA State of DevOps Report: 2023 State of DevOps Report | Google Cloud. DORA also has some guidance on monitoring and observability in their capabilities.
Google Cloud new product: Personalized Service Health
Peco and Ernest’s LinkedIn Learning course DevOps Foundations: Monitoring and Observability
Monitoring Your AI Stack (Blog): https://dtdg.co/3Qpn76g
Executive Order on Safe, Secure, and Trustworthy AI
New Relic Observability Forecast: https://newrelic.com/observability-forecast/2023/state-of-observability
CatchPoint 2023 SRE Report
We've had the folks from Honeycomb on the podcast a couple of times to talk about Observability: Christine Yen and George, Charity, and Liz
Tools list https://openapm.io/landscape organized by functionality

PagerDuty Community Update, January 19 2024

Mandi Walls — Fri, 19 Jan 2024 20:25:18 +0000

It’s update time!

Our newest release of the PagerDuty application for ServiceNow is live! Check out the details and watch the demo roundup here

Additionally, Intelligent Dashboards are now officially SUNSET. You’ll find your links to the old dashboards redirected to the new Dashboard features (find them under Analytics->Dashboard in your account). This sunset is another big step towards consolidating the Analytics Information Architecture from a UX perspective and the tech stack from an engineering perspective so we’ll be better able to keep delivering great improvements!

✍️ On the Blog and In the News

Automating the Enterprise

Jonathan Rende, SVP and GM of Product, published a new piece at Forbes discussing some of the challenges large organizations have when creating and consuming automation effectively.

APAC Retrospective

Part three in our APAC retrospective is now available. If you missed the first two installments they are here and here.

🎟️ Find us in person!

Are you making your 2024 event plans? We have a few things on the books and more to come!

Kat will be in Boston on February 8th to speak at Hiver’s event “AI vs Human: The Future of Customer Service in 2024.” Register today for this free event and please come say hi if you’re there!
Mandi will be at Tech Show London in March to chat about DevOps and AI
Then she’ll be back in Pasadena for DevOpsDays LA to talk about Chaos Engineering and Failure Fridays.

Find more Dutonians at our global events. 👋 Say hi if you’re there and share your experience with us - we’re happy to connect and spread the Pagey love!

🎙️ We’re Virtual

Chaos Carnival returns! Mandi will present Plan for Unplanned Work: Game Days and Chaos Engineering on January 24th.
Mandi will be back at Conf42 DevOps on January 25 to present Reduce Toil by Improving Automation as well as a panel on DevOps trends.
Also on the 25th, Mandi will join the folks at Cloudsoft for a panel on Demonstrating the Value of Site Reliability Engineering. Join us!

ICYMI, check out the Demo Roundup that aired on December 8 featuring six of PagerDuty’s product managers showing off new (or coming soon) features like global alert grouping, looping automations, and updates to PagerDuty’s integration into ServiceNow.

🎧 On the Podcast(s)

This week on the podcast, we were joined by Heitor Lessa, Chief Architect at AWS, for an insightful conversation. Heitor provided valuable insights on the evolution of Powertools for AWS Lambda and the strategies for fostering and nurturing an open source community. If you're interested in open source and seeking advice on community building, don't miss this episode.

Coming up later this month, we’ll have a new book club episode! These will run once a month and will feature a book that is related to operations, software development, computing, or tech in general. Our first pick is Close to the Machine by Ellen Ullman. We’re looking for readers for future episodes. If you’re interested, fill out our form

We’ll still be running our regular episodes featuring interesting folks in tech, so make sure you’re subscribed so you don’t miss anything!

In other podcast news, Mandi joined env0 Co-founder and CEO Ohad Maislish for an episode of the Infrastructure as Code podcast. Download to your favorite podcast app or check it out online.

You can now check out the new podcast version of The Unplanned Show on Spotify or Apple Podcasts. This week’s episode featured PagerDuty’s Matt Linebarger from the Customer Success Engineering team with 10 things every PagerDuty user should know.

📺 On Our Live Streams

We’re on Twitch, covering all things PagerDuty, with some of our streams also broadcasting on LinkedIn!

Our regular shows are:

The Unplanned Show with Dormain! Mondays at 4pm Eastern / 1pm Pacific (check other timezones here). Catch up on past episodes in this YouTube playlist. Upcoming episode guests include:
- January 22: Nora Jones
- February 5: Jonathan Rende
- February 26: Bryon Kroger
Terraform Time! Wednesdays at 4pm Eastern / 1pm Pacific with José Antonio! (check other timezones here).
HowTo Happy Hour: Fridays at 4pm Eastern / 1pm Pacific (check other timezones here) and now also on YouTube.

ICYMI

On Tuesday, our product folks joined us for a panel discussion around Event-Driven Automation: what it is, what it does, and how it can help you. We’ll have Frank Emery, Sam Ferguson, and Peco Karayanev on the panel. Catch the recording on YouTube.

On Thursday, PagerDuty’s CIO, Eric Johnson, and CISO, Heather Hinton, shared their 2024 predictions. Watch a replay on YouTube.

Make sure you’re following us so you can be there live! You can view our most recent episodes on Twitch and LinkedIn. See some of our earlier streams on the YouTube playlist.

🤝 Get in Touch

Join the PagerDuty Community and connect with super users to learn best practices, network, ask questions, share your thoughts, showcase your projects, give product feedback and more!

Would you like to be on our podcast, Twitch live, blog or next community meetup? Login/sign up to PagerDuty Community and submit your work here, or email us at community-team@pagerduty.com. We’d love to hear what you’re doing with PagerDuty!

💚 Follow us

Dormain: Twitter and LinkedIn
Kat: Twitter and LinkedIn
Mandi: Mastodon and Bluesky
Tara: Twitter and Mastodon
Tiago: Twitter and LinkedIn
Xenda: LinkedIn
PagerDuty on X

Retrieving Service Graph Relationships via the PagerDuty REST API

Mandi Walls — Thu, 18 Jan 2024 23:59:01 +0000

We recently had an interesting question on the PagerDuty Community forums regarding how to retrieve the list of services that make up a service graph using the PagerDuty REST API.

Service graphs can be made up of a mix of technical and business services, called dependencies, but the most common starting place for them is with a business service. How you organize your business service is really a matter of what makes sense for your teams.

There is no single endpoint query that will dump an entire service graph supporting a business service, so to get access to the whole graph, you’ll need to walk the graph.

Sample Graph

In my test account I have a set of services that make up my “Fabulous Shop”. Fabulous Shop is my business service. It is supported by another business service, Shipping Service. Fabulous shop has 5 technical dependencies: a frontend, a search backend, a cache, a shopping cart, and a database. The cache and the shopping cart both depend directly on the database. The service graph looks like this:

Retrieving Information from the REST API

I’m going to use a global API key in this example. You can also use Scoped OAuth, just make sure you’re watching which scopes are needed. They’ll be listed in the API docs for each endpoint.

Let’s investigate some of the endpoints before we get into how to tackle the service graph.

Business Services vs Technical Services

Business services are a separate kind of object in the REST API. They don’t contain the same information as technical services, so they don’t have the same object schema. Because of that, they also aren’t available behind the same endpoints. Business services will have their own set of endpoints for retrieving information, and traversing the dependencies in a service graph will require looking at both types of services independently.

This can definitely be confusing! Hopefully the code will help clear things up a bit.

Business Service ID

To get information about a business service, I need to have the object ID of that business service. You can find the ID of the service you want in a couple of ways; the easiest way is to find the service in the web interface and note the object ID in the URL. It will look something like:

https://mysubdomain.pagerduty.com/business-services/PXXXXXX

You can also make a request to the /business_services endpoint and find the service you want in the data that is returned.

First Level Dependencies

To find the first level dependencies of a business service, we’ll use the Service Dependencies endpoints - there are two! One endpoint for the dependencies of business services, and one endpoint for the dependencies of business services. If we send a request to a dependencies endpoint with an ID of the wrong service type, the API will return an error.

To find the dependencies of my business service, I make a request to https://api.pagerduty.com/business_services/PXXXXXX using my business service ID.

The first request to the business service dependencies endpoint returns this json data:

{
  "relationships": [
    {
      "dependent_service": {
        "id": "PFABSHOP",
        "relationships": null,
        "type": "business_service_reference"
      },
      "id": "DPXXXXXXXXXXXXXXXX",
      "supporting_service": {
        "id": "PSHIPSRV",
        "relationships": null,
        "type": "business_service_reference"
      },
      "type": "service_dependency"
    },
    {
      "dependent_service": {
        "id": "PFABSHOP",
        "relationships": null,
        "type": "business_service_reference"
      },
      "id": "DPXXXXXXXXXXXXXXXX",
      "supporting_service": {
        "id": "PFABFE",
        "relationships": null,
        "type": "technical_service_reference"
      },
      "type": "service_dependency"
    }
  ]
}

Let’s break it down:

The top level is an array named relationships
Each relationship in the array has four component objects:
- dependent_service with id, relationships, and type
- id of the relationship itself!
- supporting_service with id, relationships, and type
- The object type, in this case service_dependency

The dependent_service is the service that is relying on the other service, and the supporting_service is the service that is being depended upon. When looking at my service graph, the dependent_services will be above the supporting_services.

If a business service has no dependencies, the data returned will be an empty relationships array, not an error. As long as the ID is correct, you’ll get regular data.

Now we run up against some of the limitations of this endpoint. First, it’s only the direct dependencies of the first service, not an exhaustive list of all dependencies in the graph. Second, it returns only the object IDs for those services. That’s probably not the most helpful information if I want to present this information to humans, but it will be helpful for making more API requests.

Second Level Dependencies

My example business service has two direct dependencies: one is another business service, and the other is a technical service.

To find the dependencies of those services, I’ll need to make requests to two different endpoints.

Another request to /business_services/ID using the ID of the business service in the first array object - PSHIPSRV
A request to /technical_services/ID using the ID of the technical service in the second array object - PFABFE

Fortunately, the data returned from both of these endpoints has the same schema, so accessing the pieces of the objects will be similar. However, when we reach a layer inside the graph, there will be relationships for the requested service in both directions. For example, when I request the service dependencies for Fabulous Shop Frontend, my relationships object will include the relationship back to the Fabulous Shop business service. I already have that information, so as I build my methods to traverse the graph (this is starting to look like it needs recursion!), I want to leave out that relationship - otherwise my code will make the same requests over and over again forever.

That object looks like:

{
  "relationships": [
    {
      "dependent_service": {
        "id": "PFABSHOP",
        "type": "business_service_reference"
      },
      "id": "DPXXXXXXXXXXXXXXXX",
      "supporting_service": {
        "id": "PFABFE",
        "type": "technical_service_reference"
      },
      "type": "service_dependency"
    },

The other relationships for FABFE are

    {
      "dependent_service": {
        "id": "PFABFE",
        "type": "technical_service_reference"
      },
      "id": "DPXXXXXXXXXXXXXXXX",
      "supporting_service": {
        "id": "PFABCART",
        "type": "technical_service_reference"
      },
      "type": "service_dependency"
    },
    {
      "dependent_service": {
        "id": "PFABFE",
        "type": "technical_service_reference"
      },
      "id": "DPXXXXXXXXXXXXXXXX",
      "supporting_service": {
        "id": "PSRCHBE",
        "type": "technical_service_reference"
      },
      "type": "service_dependency"
    }
  ]
}

When I am traversing the relationships for a particular service, I will want to ignore any relationships where the current service is a supporting service - those relationships will document the service’s relationship with other services “above” it in the graph.

Other Helpful Bits

Depending on what you want the output of your requests to be for your users, these requests might be enough to get going. I want output that looks more like this:

Fabulous Shop (Business Service)
  Shipping Service (Business Service)
  Fabulous Shop Frontend (Technical Service)
    Fabulous Shop Shopping Cart Backend (Technical Service)
      Fabulous Shop Database (Technical Service)
    Fabulous Shop Search Service Backend (Technical Service)
      Fabulous Shop Search Caching (Technical Service)
        Fabulous Shop Database (Technical Service)

To get the service names of each service, I’ll have to make more requests, one for each service.

HOWEVER.

Getting information about a service, like the service name, will require a different endpoint for business services and technical services. Unfortunately, these requests won’t return the same object schema.

To find the name of a technical service, use the endpoint /services/ID with the service ID. The service name will be returned in the service.name subkey.

For business services, the endpoint is /business_services/ID using the service ID. The name of the service will be in the business_service.name subkey.

Put It Together

To traverse my graph, I’m going to use what is basically a Depth-First Search, except that I am interested in the arms of the graphs, not necessarily the nodes. I want to know if a node is included more than once - for example, the Fabulous Shop Database is a dependency of both the Shopping Cart and Caching services. So I don’t have to note that I’ve already seen a particular node the way you might in regular DFS.

I need to be able to make four kinds of requests:

Get relationships for a business service
Get relationships for a technical service
Get the service name for a business service
Get the service name for a technical service

I’m going to work in Python, using the requests package to make dealing with the JSON a little more straightforward.

You can find the full source code in my Github at https://github.com/lnxchk/pdgarage-samples/blob/main/python/find_dependent_services.py

Initialization

Set up. I’m importing os to access my API key from the environment, sys to read the business service ID from the command line, and requests to make the API requests.

import os
import sys
import requests

# auth
# find the api tokens in your account /api-keys
# to create a new key, you'll need to be a "manager" or "owner"
api_token = os.environ['PD_API_KEY']

Globals. To print the services out with indentation, track the indent_level. I’m also creating a dictionary to store service names so I won’t have to request a name I’ve already seen.

indent_level = 0
names_dict = {}

def print_indent():
    print("  " * indent_level, end="")

Functions

Set up a function for making requests. See the API documentation for more info on these headers.

def make_req(endpoint):
    url = "https://api.pagerduty.com/{}".format(endpoint)
    headers = {"Accept": "application/vnd.pagerduty+json;version=2",
           "Authorization": "Token token={}".format(api_token),
           "Content-Type": "application/json"}
    return requests.get(url, headers=headers)

This function will make a request to find the dependencies of a business service. It reads the indentation level, then looks for the service name - this is all for output, so can be changed for the output you prefer.

Output the service name and note it is a business service, then get the dependencies of this service.

def get_biz_deps(serv_id):
    global indent_level
    if serv_id in names_dict.keys():
        my_name = names_dict[serv_id]
    else:
        my_name = get_biz_serv_name(serv_id)
    print_indent()
    print("{} (Business Service)".format(my_name))
    indent_level += 1
    endpoint = "service_dependencies/business_services/{}".format(serv_id)
    my_deps = make_req(endpoint)

Read the data from the dependencies request. This will be a relationships array, and we want to know about only the relationships where the current service is a dependent_service. We’ll skip the relationships where the current service is a supporting_service; we’ve already seen those. They’re “above” this service in the graph.

For each dependent service, call the appropriate function to find that service’s dependencies.

    data = my_deps.json()
    for relation in data['relationships']:
        if relation['supporting_service']['id'] == serv_id:
            continue
        if relation['supporting_service']['type'] == "business_service_reference":
            get_biz_deps(relation['supporting_service']['id'])
        elif relation['supporting_service']['type'] == "technical_service_reference":
            get_tech_deps(relation['supporting_service']['id'])
    indent_level -= 1

This function works the same way as get_biz_deps, but it requires different endpoints.

def get_tech_deps(serv_id):
    global indent_level
    if serv_id in names_dict.keys():
        my_name = names_dict[serv_id]
    else:
        my_name = get_tech_serv_name(serv_id)
    print_indent()
    print("{} (Technical Service)".format(my_name))
    indent_level += 1
    endpoint = "service_dependencies/technical_services/{}".format(serv_id)
    my_deps = make_req(endpoint)
    data = my_deps.json()
    for relation in data['relationships']:
        if relation['supporting_service']['id'] == serv_id:
            continue
        if relation['supporting_service']['type'] == "business_service_reference":
            get_biz_deps(relation['supporting_service']['id'])
        elif relation['supporting_service']['type'] == "technical_service_reference":
            get_tech_deps(relation['supporting_service']['id'])
    indent_level -= 1

Functions to retrieve the names of the services. Populate the dictionary.

def get_tech_serv_name(id):
    endpoint = "services/{}".format(id)
    this_service_resp = make_req(endpoint)
    this_service = this_service_resp.json()
    names_dict[id] = this_service['service']['name']
    return(this_service['service']['name'])


def get_biz_serv_name(id):
    endpoint = "business_services/{}".format(id)
    this_service_resp = make_req(endpoint)
    this_service = this_service_resp.json()
    names_dict[id] = this_service['business_service']['name']
    return(this_service['business_service']['name'])

Main loop. Start the process by calling get_biz_deps on the service you want to explore.

if __name__ == '__main__':
    # you can pass the service ID on the command line or 
    # enter it at the prompt
    if len(sys.argv) < 2:
        this_service = input("Which service? ")
    else:
        this_service = str(sys.argv[1])
    get_biz_deps(this_service)

Summary

Thanks to our community member Olivia Mo for this question! If you have questions about PagerDuty or the PagerDuty API, join our community forums and we’ll do our best to help.

PagerDuty Community Update, January 5 2024

Mandi Walls — Fri, 05 Jan 2024 19:46:36 +0000

How Can We Help?

What would you like to see more of from the PagerDuty Community in 2024? How can we help you be more successful with PagerDuty and all of the components of the Operations Cloud? Take our super short survey and let us know!

✍️ On the Blog

Don’t be afraid of on-call!

Xenda asked some folks in the PagerDuty community how they cope with on-call responsibilities and helping their teams not fear being on-call. If you’re heading into your very first on-call rotation, you are not alone! These folks have been there and have some great tips for you.

Going beyond MTTR

MTTR is a great starting place for teams just starting out on their Incident Response and Incident Management journey. But if your team has been working on IR for a while, you might need a bit more than MTTR can give you. Google and others have written about some of the weaknesses of MTTR, and Mandi took a look at where you can get better insights in your PagerDuty account. Read more in this article.

🎟️ Find us in person!

Are you making your 2024 event plans? We have a few things on the books and more to come!

Mandi will be at Tech Show London in March to chat about DevOps and AI
Then she’ll be back in Pasadena for DevOpsDays LA to talk about Chaos Engineering and Failure Fridays.

Find more Dutonians at our global events. 👋 Say hi if you’re there and share your experience with us - we’re happy to connect and spread the Pagey love!

🎙️ We’re Virtual

NEXT WEEK Mark your calendars for the next Terraform Roundtable! January 9 at 11am Pacific / 2pm Eastern / 8pm Central Europe. Join us to chat all things PagerDuty and Terraform!

Chaos Carnival returns! Mandi will present Plan for Unplanned Work: Game Days and Chaos Engineering on January 24th.
Mandi will be back at Conf42 DevOps on January 25 to present Reduce Toil by Improving Automation.

🎧 On the Podcast(s)

We’ve kicked off 2024 in style with an episode featuring Abby Bangser. Abby’s one of the co-chairs of the CNCF’s Platforms Working Group. She joined us to talk about all things platform engineering!

We’ll still be running our regular episodes featuring interesting folks in tech. Make sure you’re subscribed so you don’t miss anything!

📺 On Our Live Streams

We’re on Twitch, covering all things PagerDuty, with some of our streams also broadcasting on LinkedIn!

Our regular shows are:

The Unplanned Show with Dormain! Mondays at 4pm Eastern / 1pm Pacific (check other timezones here). Catch up on past episodes in this YouTube playlist. Prefer an audio-only podcast? Check out the new podcast version of The Unplanned Show on Spotify or Apple Podcasts
Upcoming episodes:
- January 8: Manu Gurudatha on “From Slipping to Shipping”
Terraform Time! Wednesdays at 4pm Eastern / 1pm Pacific with José Antonio! (check other timezones here).
HowTo Happy Hour: Fridays at 4pm Eastern / 1pm Pacific (check other timezones here) and now also on YouTube.
- More from our product team on new Operational Maturity reports will be coming soon! Plus, Frank returns with more tips on using Event Orchestration!
January 16, join our product folks for a panel discussion around Event-Driven Automation: what it is, what it does, and how it can help you. We’ll have Frank Emery, Sam Ferguson, and Peco Karayanev on the panel. Sign up on LinkedIn.

Make sure you’re following us so you can be there live! You can view our most recent episodes on Twitch and LinkedIn. See some of our earlier streams on the YouTube playlist.

🤝 Get in Touch

Join the PagerDuty Community and connect with super users to learn best practices, network, ask questions, share your thoughts, showcase your projects, give product feedback and more!

💚 Follow us

Dormain: Twitter and LinkedIn
Kat: Twitter and LinkedIn
Mandi: Mastodon and Bluesky
Tara: Twitter and Mastodon
Tiago: Twitter and LinkedIn
Xenda: LinkedIn
PagerDuty on X

PagerDuty Community Update December 15, 2023

Mandi Walls — Fri, 15 Dec 2023 20:07:40 +0000

How Can We Help?

✍️In the News

Congratulations to PagerDuty CEO Jenn Tejada, Chief People Officer Joe Militello, and Senior Director of Data Science Mitra Goswami for being recognized as Inspiring Leaders by Inspiring Workplaces Awards!
// Detect dark theme var iframe = document.getElementById('tweet-1734325375864308143-10'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1734325375864308143&theme=dark" }

Also, you can vote for PagerDuty Process Automation in the DevOps Dozen Awards! Open until Dec 31!

🎟️ Find us in person!

Are you making your 2024 event plans? We have a few things on the books and more to come!

Mandi will be at Tech Show London in March to chat about DevOps and AI
Then she’ll be back in Pasadena for DevOpsDays LA to talk about Chaos Engineering and Failure Fridays.

Find more Dutonians at our global events. 👋 Say hi if you’re there and share your experience with us - we’re happy to connect and spread the Pagey love!

🎙️ We’re Virtual

Mark your calendars for the next Terraform Roundtable! January 9 at 11am Pacific / 2pm Eastern / 8pm Central Europe. Join us to chat all things PagerDuty and Terraform!

Chaos Carnival returns! Mandi will present Plan for Unplanned Work: Game Days and Chaos Engineering on January 24th.

🎧 On the Podcast(s)

Python in Space. Last week, Mandi spoke with Mike Fiedler, a Security and Safety Engineer for PyPI, about various use cases for the Python programming language and the importance of ensuring their security. They also discussed the challenges and benefits of working in the Open Source community and even touched upon the topic of Python being used in space missions.

Next week will be our final episode of 2023, and our year in review!

Plus, we have a new feature coming up in January, so make sure you’re subscribed so you don’t miss it!

📺 On Our Live Streams

We’re on Twitch, covering all things PagerDuty, with some of our streams also broadcasting on LinkedIn!

Our regular shows are:

The Unplanned Show with Dormain! Mondays at 4pm Eastern / 1pm Pacific (check other timezones here). Catch up on past episodes in this YouTube playlist. Prefer an audio-only podcast? Check out the new podcast version of The Unplanned Show on Spotify or Apple Podcasts
Upcoming episodes:
- December 18; Todd Whitney on zero maintenance window updates on YouTube or LinkedIn
- January 8: Manu Gurudatha on “From Slipping to Shipping”
Terraform Time! Wednesdays at 4pm Eastern / 1pm Pacific with José Antonio! (check other timezones here).
HowTo Happy Hour: Fridays at 4pm Eastern / 1pm Pacific (check other timezones here) and now also on YouTube.
- Coming in January: A panel discussion focused on Event-Driven Automation and more from our product team on new Operational Maturity reports! Plus, Frank returns with more tips on using Event Orchestration!

Earlier today we wrapped up our 2023 streams. You can view the recording on YouTube.

Make sure you’re following us so you can be there live! You can view our most recent episodes on Twitch and LinkedIn. See some of our earlier streams on the YouTube playlist.

🤝 Get in Touch

Join the PagerDuty Community and connect with super users to learn best practices, network, ask questions, share your thoughts, showcase your projects, give product feedback and more!

💚 Follow us

Dormain: Twitter and LinkedIn
Kat: Twitter and LinkedIn
Mandi: Mastodon and Bluesky
Tara: Twitter and Mastodon
Tiago: Twitter and LinkedIn
Xenda: LinkedIn
PagerDuty on X

PagerDuty Community Update, November 10 2023

Mandi Walls — Fri, 10 Nov 2023 18:51:04 +0000

Backstage with PagerDuty

Are you using Backstage for your internal developer portal? PagerDuty is now on the Spotify Marketplace for Backstage! Trigger incidents, see active incidents, and identify the on-call engineer for your services directly from Backstage. Check it out here and hang out with us on the Backstage Discord server!

🎟️ Find us in person!

Where in the world is the PagerDuty DevRel team?

Tiago spent this week at KubeCon + CloudNativeCon North America. Where you there too? Did you meet up with any of the PagerDuty folks? Let us know in the comments! // Detect dark theme var iframe = document.getElementById('tweet-1722325067042873390-873'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1722325067042873390&theme=dark" }
Kat spoke yesterday at DevOps Barcelona. Did you catch her session Mastering Incident Communication?

Where else you can find Dutonians in November:

Nov 23: London: Women of Silicon Roundabout Conference
Nov 23 to Dec 1: AWS re:Invent 2023 Las Vegas. Come say hi! We’re booth #1096, and we’ll be running content in our booth theater all week!

Find more Dutonians at our global events. 👋 Say hi if you’re there and share your experience with us - we’re happy to connect and spread the Pagey love!

We’re Virtual!

Get the recording of our webinar The Art of Deciding What to Automate
Sign up for GenAI Unleashed: Transforming Customer Experience in the Digital Era, running Nov 14.
On Nov 16, Mandi will be presenting Best Practices for On Call Teams at DeveloperWeek Enterprise. Click here for a free OPEN pass to DeveloperWeek Enterprise!
Mandi will also be presenting Plan for Unplanned Work: Game Days and Chaos Engineering at Pozitive Technologies on Nov 23.
Plan ahead and mark your calendars for the next Terraform Roundtable! January 9 at 11am Pacific / 2pm Eastern / 8pm Central Europe. Join us to chat all things PagerDuty and Terraform!

🎧 On the Podcast(s)

Milestone week on Page it to the Limit! Our 100th episode! We interviewed some of the most-tenured PagerDuty employees to learn more about how PagerDuty has changed and evolved over the last 14 years. If you’ve never heard PagerDuty’s origin story, or how those awesome barbershop quartet ringtones were created, those stories and more are in this episode!

New on The Unplanned Show, Episode 5: DataOps with Snowflake!

Subscribe in your favorite podcatcher!

📺 On Our Live Streams

We’re on Twitch, covering all things PagerDuty, with some of our streams also broadcasting on LinkedIn!

Our regular shows are:

The Unplanned Show with Dormain! Mondays at 4pm Eastern / 1pm Pacific (check other timezones here). Catch up on past episodes in this YouTube playlist. Prefer an audio-only podcast? Check out the new podcast version of The Unplanned Show on Spotify. Upcoming episodes:
- November 13: LLM Observability with Charity Majors and James Governor
- November 20: Cultures of Automation with Jamie Vernon
Terraform Time! Wednesdays at 4pm Eastern / 1pm Pacific with José Antonio! (check other timezones here).
HowTo Happy Hour: Fridays at 4pm Eastern / 1pm Pacific (check other timezones here) and now also on YouTube.
- Upcoming episodes: Friday, November 17th we’re talking Analytics with Anojan Gunasekaran.

Make sure you’re following us so you can be there live! You can view our most recent episodes on Twitch and LinkedIn. See some of our earlier streams on the YouTube playlist.

🤝 Get in Touch

Join the PagerDuty Community and connect with super users to learn best practices, network, ask questions, share your thoughts, showcase your projects, give product feedback and more!

💚 Follow us

Dormain: Twitter and LinkedIn
Kat: Twitter and LinkedIn
Mandi: LinkedIn and Bluesky
Tara: Twitter and Mastodon
Tiago: Twitter and LinkedIn
Xenda: LinkedIn
PagerDuty on X

PagerDuty Community Update, October 20 2023

Mandi Walls — Fri, 20 Oct 2023 19:23:54 +0000

PagerDuty’s Getting a New Look!

Check it out! PagerDuty’s main website has a fresh new design! Find out how the superstars on our brand team put it all together!

📝 Take the PagerDuty API Survey

We want to hear from you! Tell us how familiar you are with different PagerDuty APIs and integrations, as well as what (if anything) is preventing you from using them. Login to the PagerDuty Community Forums and take the PagerDuty API survey challenge.

📺 Managing Major Incident Communications Channels

Your chat platform / internal text communication method of choice can help responder teams and stakeholders manage comms during and after incidents. There’s lots of ways to organize incident communications, depending on your organization and workflows. PagerDuty uses Slack and we have a number of different channels for managing the communication and coordination of incidents.

🎟️ Find us in person!

Where in the world is PagerDuty DevRel? The next few months are packed with PagerDuty Advocates worldwide in Taiwan and London!

Did you see Dormain at Gartner Symposium in Orlando? She gave a talk titled 5 mistakes CIOs make with GenAI, AIOps, and Automation.

Other Dutonians you’ll find out and about:

Earlier this week, PagerDuty's Senior Director of Data Science Mitra Goswani presented Expedition to the LLM And AI Frontier to LeadDev West Coast. Mitra was a guest at the Unplanned Show last August and spoke about learnings from building with large language models (LLMs). Watch here 👈

Sanghamitra Goswami on LinkedIn: #ai

PagerDuty #AI squad at LeadDev Rebecca Larsen

linkedin.com
Everaldo Aguiar (Senior Engineering Manager) will represent PagerDuty at AIDevWorld in Santa Clara (CA), Oct 24-26, with AI in AIOps: Minimizing Disruption and Accelerating Incident Resolution.

Find more Dutonians at our global events. 👋 Come say hi if you’re there and share your experience with us - we’re happy to connect and spread the Pagey love!

We’re Virtual!

Mandi will be presenting Futuristic Luxury Incident Response at All Day DevOps on Oct 26. Register to get 24 hours of DevOps!
Join us November 2nd for a panel discussion on Modern Monitoring and Observability, featuring Ernest Mueller of Accenture and Peco Karayanev PagerDuty, authors of DevOps Foundations: Monitoring and Observability on LinkedIn Learning, plus Jason Hand of Datadog and Steve McGhee of Google!
In November, Mandi will be presenting Best Practices for On Call Teams at DeveloperWeek Enterprise. Click here for a free OPEN pass to DeveloperWeek Enterprise!
She’ll also be presenting Plan for Unplanned Work: Game Days and Chaos Engineering at Pozitive Technologies on November 23.

🎧 On the Podcast(s)

This week Mandi talked to JJ Asghar of IBM about whether you really need to devolve that monolith into microservices, or if you should just leave it alone. Listen online now!

Subscribe to Page It To the Limit so you never miss an episode! We’ve got a bunch of great episodes in the pipeline for you, including our 100th episode on November 7th!

📺 On Our Live Streams

We’re on Twitch, covering all things PagerDuty, with some of our streams also broadcasting on LinkedIn!

Our regular shows are:

The Unplanned Show with Dormain! Mondays at 4pm Eastern / 1pm Pacific (check other timezones here). Catch up on past episodes in this YouTube playlist.
Upcoming episode:
- October 23: Meg Watson (Product Lead @ Spotify) & Dev Advocate Tiago Barbosa on PagerDuty plug-in for Backstage on Twitch, LinkedIn, or YouTube
Terraform Time! Wednesdays at 4pm Eastern / 1pm Pacific with José Antonio! (check other timezones here)
HowTo Happy Hour: Fridays at 4pm Eastern / 1pm Pacific (check other timezones here)
Upcoming guests:
- Postponed Friday, October 20, Mya King joins us to talk about what’s new in the PagerDuty Slack integration. We’ll let you know the new date soon!
- Friday, October 27, Alex Quintana will join the stream to talk about what’s new in Onboarding and Operational Maturity!
- Coming in November, new features in Analytics!

Make sure you’re following us so you can be there live! You can view our most recent episodes on Twitch and LinkedIn. See some of our earlier streams on the YouTube playlist.

🤝 Get in Touch

Join the PagerDuty Community and connect with super users to learn best practices, network, ask questions, share your thoughts, showcase your projects, give product feedback and more!

Would you like to be on our podcast, Twitch live, blog or next community meetup? Login/sign up to PagerDuty Community and submit or work here, or email us at community-team@pagerduty.com. We’d love to hear what you’re doing with PagerDuty!

💚 Follow us on Social Media!

Dormain: Twitter and LinkedIn
Kat: Twitter and LinkedIn
Mandi: Twitter and Bluesky
Tara: Twitter and Mastodon
Tiago: Twitter and LinkedIn
Xenda: LinkedIn
PagerDuty on X

Managing Major Incident Communications Channels

Mandi Walls — Tue, 17 Oct 2023 18:02:27 +0000

Figuring out how to mobilize and organize multiple teams to respond to a major incident can take some trial and error. Unfortunately, errors can hurt your business and lead to longer resolution times.

You can mobilize responders and keep stakeholders informed using PagerDuty’s Incident Workflows, but how do you decide what actions to add to your workflow? We’ll provide some examples here based on how PagerDuty does PagerDuty for Major Incident Response.

Anyone at PagerDuty can declare a Major Incident, based on an existing incident. Often, these are initiated when a single team has investigated an issue on their services and found the behaviors impact or derive from other services, and other teams need to investigate. Major Incidents can also be declared directly when someone finds a problem or something is reported by a customer.

Set Up Priorities

Some incidents are more important than others. Your organization should work toward a universal understanding of what constitutes a major incident for your environment. An incident might be classified as “major” if it impacts a certain number of customers, reaches a predetermined duration, includes an important function, or any number of other criteria. You can use incident priorities to classify incidents within your PagerDuty account, and create different workflows based on your defined Priority or Severity categories.

It’s ok to be wrong about the priority of an incident while it’s happening, but it’s better to over-respond rather than under-respond. We refer to this as never hesitate to escalate.

Communications Channels

Getting responders to the right place for a major incident is key to keeping your response and resolution times low. Responders shouldn’t be confused about where the activities are taking place or how to find the right channel in your chat application.

For major incidents, using a single, permanent channel to coordinate the response will help all of the responders in your organization know exactly where to go when they are added to an incident. Think of this channel as your predetermined meeting place for an emergency drill.

In PagerDuty, you can add a new channel to an incident using Incident Workflows. We have an Incident Workflow titled “Major Incident Workflow” for exactly this purpose. When someone determines that an incident has reached SEV-2 or SEV-1 based on our criteria, a responder can run the Incident Workflow, and several actions take place:

Additional responders are added to the incident, including our customer liaisons.
Stakeholders are added to the incident. These folks have opted in to being notified, as well as leadership, customer-facing teams, and others who might need to stay informed of an incident, but who won’t be actively working on the response.
A Slack channel is added. This is our major incident room, and it is the same channel for every Major Incident. Other incidents might have automatically-generated channels, but Major Incidents are always handled in this channel.

The predictability of this method helps responders to major incidents get to the right place fast.

Communicating with Stakeholders

Not everyone who wants to know about an incident wants to be on the always-notify stakeholder list included in the Incident Workflow above. Some folks in the organization might only be interested in an incident if it touches on their services, or a group of services. There also might just be curious folks who want to know what is going on. That’s great! But they probably aren’t going to get much helpful information hanging out in the response channel. For these folks, we have a read-only incident-updates channel.

While a Major Incident is running, any updates added to the incident will go to the specified Slack channel. In the setup to integrate PagerDuty with Slack, you can set up these types of channels and decide what kind of updates the channel should receive. You can have any number of these connections for your Slack workspace.

Like the major incident room, the incident updates channel is always used for status updates to major incidents, so stakeholders can always find it.

After the Incident

When a major incident is resolved, there are a number of activities that responders need to complete, including creating documents and setting a time for the post incident review meeting.

To help teams coordinate, a primary channel exists, incident-followup. Responders can post the draft of the postmortem report, any work tickets, and other information here for everyone to find. The team responsible for the post-incident review can then create an incident-specific channel to discuss the incident and deliverables. That channel is archived after the incident review is completed.

Communication for the Right People at the Right Time

The mixture of permanent and short-lived channels allows responders to mobilize effectively and collaborate for the duration of the incident and post-incident activities.

Using static channels accelerates mobilization for your most important incidents, gives stakeholders a place to follow along for updates, and provides information for everyone in the organization.

For more on how to add Slack or Microsoft Teams to your incident response workflows, see our Knowledge Base. To learn more about our Incident Response process, check out our Ops Guide. Join our community to connect with and learn from other PagerDuty users.

The Ops Community ⚙️: Mandi Walls

Resources for Tackling Alert Fatigue with SLOs, Automation, and Machine Learning

It's almost time for re:Invent

It's DRY

It's Pricey

It's Packed

It's Far

What are your survival tips?

Resources for Best Practices for On Call Teams

Resources for Plan for Unplanned Work: Game Days and Chaos Engineering

Resources: Monitoring and Observability Panel Discussion

PagerDuty Community Update, January 19 2024

It’s update time!

✍️ On the Blog and In the News

Automating the Enterprise

APAC Retrospective

🎟️ Find us in person!

🎙️ We’re Virtual

🎧 On the Podcast(s)

📺 On Our Live Streams

ICYMI

🤝 Get in Touch

💚 Follow us

Retrieving Service Graph Relationships via the PagerDuty REST API

Sample Graph

Retrieving Information from the REST API

Business Services vs Technical Services

Business Service ID

First Level Dependencies

Second Level Dependencies

Other Helpful Bits

Put It Together

Initialization

Functions

Summary

PagerDuty Community Update, January 5 2024

How Can We Help?

✍️ On the Blog

Don’t be afraid of on-call!

Going beyond MTTR

🎟️ Find us in person!

🎙️ We’re Virtual

🎧 On the Podcast(s)

📺 On Our Live Streams

🤝 Get in Touch

💚 Follow us

PagerDuty Community Update December 15, 2023

How Can We Help?

✍️In the News

🎟️ Find us in person!

🎙️ We’re Virtual

🎧 On the Podcast(s)

📺 On Our Live Streams

🤝 Get in Touch

💚 Follow us

PagerDuty Community Update, November 10 2023

Backstage with PagerDuty

🎟️ Find us in person!

We’re Virtual!

🎧 On the Podcast(s)

📺 On Our Live Streams

🤝 Get in Touch

💚 Follow us

PagerDuty Community Update, October 20 2023

PagerDuty’s Getting a New Look!

📝 Take the PagerDuty API Survey

📺 Managing Major Incident Communications Channels

🎟️ Find us in person!

Sanghamitra Goswami on LinkedIn: #ai

We’re Virtual!

🎧 On the Podcast(s)

📺 On Our Live Streams

🤝 Get in Touch

💚 Follow us on Social Media!

Managing Major Incident Communications Channels

Set Up Priorities

Communications Channels

Communicating with Stakeholders

After the Incident

Communication for the Right People at the Right Time