PagerDuty’s Slack integration is a one-stop-shop for managing your incidents from your Slack channels. You can ack and resolve incidents, add responders, and run custom incident actions.
It’s pretty awesome.
But there’s a bunch of other information in your PagerDuty account that might be helpful to have in your Slack channels. Our internal teams have some integrations that do things like populate the channel description with the current oncall information, so folks who know the team name via Slack can get the info they need. These sorts of integrations make use of the PagerDuty REST API, and run as little jobs in places like AWS Lambda.
We don’t open source these, though there are other examples of similar functionality floating around. So we occasionally get questions in the community forums about keeping teams informed about what’s going on in your ecosystem. A recent question asked about notifications for maintenance windows.
The maintenance windows feature is handy for when you’re planning some downtime for a service, maybe to do an upgrade or some other work. All the alerts and incidents related to that service will be ignored until the end of the window, so you don’t annoy your team’s on-call engineer! But maintenance windows don’t have a notification feature; they appear in your service directory on their own tab, or on the page for the impacted service, and now you can manage them in the mobile app!
They aren’t included in the Slack integration. But we’ve got an API, and Slack has webhooks, so let’s see what we can do. You can download the code and use it as-is or as a starting place for your own adventures with the PagerDuty API.
Prerequisites
We need a couple of things first:
- An API key for the PagerDuty REST API. Not the events API!
- A webhook to send information to on Slack. I followed the steps here to create my own app and enable webhooks. There’s other ways to get information into Slack, but this is the most basic one. You can try out the others!
In Python
I’m going to give this example in Python (and I apologize in advance if I offend anyone’s Pythonic sensibilities ;) ). There is a client library for the PagerDuty API in Python called pdpyras that will help us set up our request for the maintenance window information. I’ll be using Python 3.9, but most of what we’re doing should work in 3.8. If you’re using an earlier version of Python, you may get an error about the datetime formats we’re using.
I’m also using the json library to manipulate the request payloads, os to read account settings out of my environment (you could use a secrets vault or other storage), requests to send the Slack request, and datetime to do date math.
Start an API Session
To set up our session with the PagerDuty API, I need to initialize it with my API key.
api_token = os.environ['PD_API_KEY']
## initialize the PagerDuty API session
session = APISession(api_token)
I’m also going to access my Slack webhook url from the environment:
slack_url = os.environ['SLACK_CHANNEL_URL']
Make the PagerDuty API Request
The endpoint we need is the /maintenance_windows endpoint. I’m also going to use ?filter=open, which will include only the windows that are happening now, and those that will run in the future. I’m going to use my session object to make the request:
endpoint = "maintenance_windows?filter=open"
response = session.rget(endpoint)
Build the Slack Message
I’m going to use the data returned from my API request to build a new message to send to Slack. The maintenance_window objects come back in their own JSON format, with some stuff I don’t really need for just these informational messages, but you might find other info in there you might find helpful in your version.
Slack messages use Block Kit for formatting. There is a helpful Block Kit Builder on the Slack dev site. I’m using a single header block as a title and a mrkdwn block for each maintenance window.
The payload object is going to be the full document that we’ll turn into JSON. The array of blocks will have one entry for each block plus the header, so I’m using .append() to build the array.
payload = ""
blocks = []
header_block = {
"type": "header",
"text": {
"type": "plain_text",
"text": "The following maintenance windows are scheduled:"
}
}
blocks.append(header_block)
If there are no windows scheduled in my account, we only have one message:
if not response:
my_block = {
"type": "section",
"text": {
"type": "mrkdwn",
"text": "There are no maintenance windows scheduled"
}
}
blocks.append(my_block)
If there are windows scheduled, walk the response object. A single maintenance window can impact multiple services, so the Slack message will have one line for each service, even if it is included in a larger window. You could change that output into something more collective if you wanted to.
In here I’m also doing the date comparisons to determine if a window is happening now, today, or tomorrow and later. For Current and Today windows, I’m adding some formatting to bold part of the message. I’m using the star object for that, because * is bold in this markdown. :)
The output messages I’m creating are pretty basic. You could add more information, or a link to the PagerDuty web UI in your messages; there’s plenty of other helpful bits in the maintenance_window object.
else:
today = datetime.today()
for mw in response:
# formatting string for making today's windows stand out
star = ""
for serv in mw['services']:
mw_id = mw['id']
desc = mw['description']
service = serv['summary']
start_time = mw['start_time']
s_time = datetime.strptime(mw['start_time'], "%Y-%m-%dT%H:%M:%S%z")
if today.date() == s_time.date():
star = "*Today* "
end_time = mw['end_time']
e_time = datetime.strptime(mw['end_time'], "%Y-%m-%dT%H:%M:%S%z")
if e_time.timestamp() > today.timestamp() > s_time.timestamp():
star = "*Happening Now:* "
my_block = {
"type": "section",
"text": {
"type": "mrkdwn",
"text": ">" + star + service + " from " + start_time + " to " + end_time
}
}
blocks.append(my_block)
Now I have a set of entries in the blocks object representing all the pieces of my message. Time to build it and turn it into JSON.
payload = {
"blocks": blocks
}
j_payload = json.dumps(payload)
Send the Slack Request
Now I’m going to send the request to my Slack webhook URL. I’m setting the Content-Type in the headers, and including the JSON payload. If all goes well, the script prints ok and I get my message in Slack:
headers = {"Content-Type": "application/json"}
sent_msg = requests.post(slack_url, headers=headers, data=j_payload)
sent_msg.raise_for_status()
print(sent_msg.text)
Output Examples
Here’s what these messages look like in action!
Case 1: No Maintenance Windows
Case 2: Some Windows Today
Case 3: Only future windows
Case 4: Currently Running Window
Case 5: Window with Multiple Services
Future Enhancements
I've made a couple of small improvements since this listing. You'll find the updated version in GitHub here
The maintenance_window object has a bunch of other interesting information in it. I've added a link to each listing for the corresponding page in the PagerDuty UI, and the service names are also links to the service page.
These timestamps are also kind of hard to read, so I've reformatted them to improve the readability of those.
So the output now looks more like this:
There are definitely places in the code that could be turned into functions to improve reuse for all of the information you might want to send from PagerDuty to Slack.
Additionally, you could use one of the other Slack application methods to create a more robust set of notifications, and send each maintenance window message to a channel with just the team that owns the services. You’d want somewhere to track the relationships between teams in PagerDuty and the channels on Slack.
I hope you found this interesting! The PagerDuty Garage is on Twitch every Monday morning at 10am Eastern / 7am Pacific. Stop by and say hi! If you have a project using the PagerDuty API, join us on the stream to show it off! Email us at community-team@pagerduty.com.
Latest comments (0)