PagerDuty Alerts for Important(ish) Stuff in GitHub

(Or wherever webhooks are found)

Our team at PagerDuty has a number of open source repositories for our Ops Guides. These are a bunch of online docs that we created and manage about topics we think will help folks who use our products. The projects are stable; they don’t get much in the way of additions, outside pull requests, or issues, which means we’re not watching them too closely. So, when something does come in, we’d like to know about it by getting a PagerDuty notification, and not risk it getting lost in all of the GitHub info we get about other, busier repos. Plus, folks have their own preferences for how they handle GitHub notifications, so a generic solution for the whole team will be helpful.

PagerDuty has GitHub integrations that make use of change events. Change events are helpful in that they can provide context if a service has an incident - was that incident related to a recent change? - but change events don’t create alerts and notifications. These integrations are awesome, just not what we want for this use case!

What We Want

Our problem is this:

We want PagerDuty notifications generated from activities - issues and pull requests - on a number of low-traffic GitHub repositories, without folks needing to change their GitHub preferences.

How We Get There

We’re going to use a construct in PagerDuty called an App Event Transformer. Don’t worry if the docs seem complex! We’re going to use some existing code to bootstrap our project!

App event transformers allow us to use JavaScript to transform (get it?) incoming information into PagerDuty events. That JavaScript is edited in the PagerDuty web UI and lives on our servers. Even some of our third party integrations are transformers under the hood. In this article we’ll be catching events from GitHub and transforming them into PagerDuty events!

Start Here!

A great starting point is the PagerDuty App Event Transformer Sample project on GitHub. This project uses GitHub as the information source that will trigger PagerDuty alerts for new issues opened in a repo.
Follow the set up instructions in the README and use the code in transformer.js to create your own app event transformer. The instructions for setting up a transformer in that project are quite detailed, so we won’t repeat them here. Follow those, and when you have that working, come back here and we’ll add the rest!

How Do You Debug This??

Fortunately for us, the debug setting in the app event transformer set up is super helpful. It will tell us if there are missing values or if the alert couldn’t be parsed for some reason. In this example, the code is trying to find a value for an href that is missing in a link object:

GitHub has great tools for managing and configuring webhooks. One of those is a feature in the webhook support for redelivering a payload, so we won’t have to create a bunch of extra issues or pull requests to test out our code. We can create one to start and use redeliver. You can see this feature from the “edit” screen of your webhook. Click the Recent Deliveries tab at the top of the page:

You’ll see the rest of the webhook, including headers, below the redeliver button. This is super useful for figuring out which parts of the payload you’d like to use in your alert. You can scroll through the attributes of the payload and choose the pieces that make the most sense for how your team works and manages incidents.

Changing the Alert Format a Little Bit

Before we add support for pull requests, let’s update the format of the output a little bit to make it more extensible. We’ll walk through those changes together, but you can also find the full source code for the updated transformer at the bottom of this post.

Inside the payload for the event the custom_details in a PagerDuty event can be almost anything you need it to be. In the current code, it’s just a string capturing the body of the GitHub issue and the username of the GitHub user who posted the issue:

   let normalized_event = {
        event_action: incidentAction,
        dedup_key: dedupKey,
        payload: {
            summary: incidentSummary,
            source: 'GitHub',
            severity: PD.Critical,
            custom_details: `${body.issue.body} (${body.issue.user.login})`
        },

        links: [{
            "href": githubLinkURL,
            "text": "View In GitHub"
        }],
    }

The issue body can have a lot of info, and different teams might find other parts of the issue object more helpful to include in the alert. So let’s change the code around a little bit to make some space for other information.

First, I’m going to add a new variable on line 21, called customDetails, so that when we add support for pull requests, we can pick different parts of the incoming payload object based on the event type:

    let customDetails = ‘’;

Now assign a value to customDetails in the section that reads issues event types. I’ve added this at line 33, inside our if statement checking to see if an action is opened or reopened:

            customDetails = {
                body: ${body.issue.body},
                user: ${body.issue.user.login}
            };

Now we want to add this customDetails variable to our payload as the value of the custom_details field,:

…
payload: {
            summary: incidentSummary,
            source: 'GitHub',
            severity: PD.Critical,
            custom_details: customDetails
        },
…

When my event is delivered to PagerDuty, I’ll have a nice table of information, and if I want to add more pieces later, it’s easy to add them to customDetails:

Adding Support for Pull Requests

With the introduction of the customDetails variable the code is now a little more flexible, and we can add support for pull requests to this same app event transformer. I’m choosing to do this with one transformer to keep things simple for the team who will be receiving these alerts. Everything we care about from GitHub will be dealt with by the same folks, so the alerts can go to the same destination. You might have other use cases where different actions should route to different teams. In those situations you can add multiple app event transformers to handle all of your use cases. Especially for Github, where the webhook settings offer a lot of granularity for what actions are sent to which webhooks, it’s handy to be able to split things up.

To start off, edit the webhook you set up in GitHub, and make sure it’s either set to Send me everything or that both issues and pull requests are selected under Let me select individual events.

Now add the code for pull requests after the section for issues. The event type in Github is pull_request, and it will work almost exactly the same as the section for issues, except that the information we want is in a different object. I’m adding this after the if block where we checked whether githubEventType was issues, around line 41. Now we’re checking if githubEventType is pull_request:

    } else if (githubEventType === 'pull_request') {
        dedupKey = `${body.pull_request.id}`; 
        if(body.action === 'opened' || body.action === 'reopened') {
            incidentSummary = `PR opened on ${body.repository.full_name}`;
            githubLinkURL = body.pull_request.html_url;
            customDetails = {
                title: `${body.pull_request.title}`,
                body: `${body.pull_request.body}`,
                user: `${body.pull_request.user.login}`
            };
        } else {
            emitEvent = false;
        }

    // if the github event type is something other than issues, don't send an event    
    } else {
        emitEvent = false;
    }

I’ve changed the incidentSummary slightly from the issues section, just as an example. I’ve added the title, body, and user to the custom details so they’ll show up in the information table in the incident. The body of this PR is empty, so it shows up as “null” in my alert:

Adjust Service Notifications

Now that I have the alerts I want, from the GitHub activities I want to hear from, I can decide how to notify the folks on my team.

These are low-priority alerts; we don’t want to miss them, but there’s no reason to wake anyone up about them, or alert on the weekends. I could set the service to “Low urgency notifications, do not escalate”, which will use the on call engineer’s account settings for low urgency alerts. Hopefully that is something quiet, like an email notification.

I could also use the support hours setting for the service that will receive the alerts. I can define a limited window in which these alerts will generate notifications to the team:

This is also a helpful option. We won’t miss things, and they won’t wake us up!

I can send alerts to this service from as many GitHub repositories as I want to. We have 10 to watch, and because the code is reading data out of the webhook payload, every repository can be configured the same way, and team members won’t have to change their GitHub preferences.

Summary

App event transformers are a powerful way to give teams control of the information they receive from important systems. If your ecosystem components aren’t supported out of the box or via an integration, a bit of JavaScript can help you get organized! If you build a transform for your team, tell us about it! We’d love to hear how you’re getting value from PagerDuty.

Learn more about PagerDuty’s features and sign up for a trial on our website.

Source Code

export function transform(PD) {
    // capture incoming webhook body
    let body = PD.inputRequest.body;
    // capture incoming webhook headers
    const headers = PD.inputRequest.headers;
    let emitEvent = true;

    // GitHub stuff
    let githubEventType = '';

    // capture the GitHub event type
    for (var i = 0; i < headers.length; i++) {
        if('x-github-event' in headers[i]) {
            githubEventType = headers[i]['x-github-event'];
        }
    }
    let githubLinkURL = '';

    // Incident stuff
    let incidentSummary = '';
    let dedupKey = '';
    let customDetails = '';
    let incidentAction = PD.Trigger;

    // only process webhook if it's an 'issues' event type
    if (githubEventType === 'issues') {
        dedupKey = `${body.issue.id}`;

        // if the github issue is opened set incident summary with the issue title
        if (body.action === 'reopened' || body.action === 'opened') {
            incidentSummary = `[${body.repository.full_name}] Issue:  ${body.issue.title}`;
            githubLinkURL = body.issue.html_url;
            customDetails = {
                body: `${body.issue.body}`,
                user: `${body.issue.user.login}`
            };
        // if neither of these actions are occurring, don't send an event   
        } else {
            emitEvent = false;
        }
     // repeat for pull requests
     } else if (githubEventType === 'pull_request') {
        dedupKey = `${body.pull_request.id}`; 
        if(body.action === 'opened' || body.action === 'reopened') {
            incidentSummary = `PR opened on ${body.repository.full_name}`;
            githubLinkURL = body.pull_request.html_url;
            customDetails = {
                title: `${body.pull_request.title}`,
                body: `${body.pull_request.body}`,
                user: `${body.pull_request.user.login}`
            };
        } else {
            emitEvent = false;
        }

    // if the github event type is something other than issues or PRs, don't send an event    
    } else {
        emitEvent = false;
    }

    // preparing the normalized_event to be sent to pagerduty
    let normalized_event = {
        event_action: incidentAction,
        dedup_key: dedupKey,
        payload: {
            summary: incidentSummary,
            source: 'GitHub',
            severity: PD.Critical,
            custom_details: customDetails
        },

        links: [{
            "href": githubLinkURL,
            "text": "View In GitHub"
        }],
    }

    if (emitEvent) PD.emitEventsV2([normalized_event])
  }