Grafana alerting notifications in Mattermost

Grafana is a tool for monitoring whatever you want. Sometimes when the stuff you monitor behaves weirdly or goes all haywire, you want a notification. You have set up your Grafana alert and a notification policy that sends an email to your grandmother.

That Ol' Gran gets her information is great, but you are looking for something more modern that fits the 2020's lifestyle. You want something that is more like a frappucino rather than a black coffee with milk.

Your team uses Mattermost, which is an open source alternative to Slack with the option to self-host. This is the modern way to communicate. Here you already have some bots informing you about your CI/CD pipelines, curing your loneliness and one that pings you when Wheatley logs in. So adding another shouldn't be that hard.

You think to yourself:

Mattermost and Slack looks the same and acts the same, so there must be a Grafana contact point option for Mattermost too

In your quest to the solution you do a quick Google search. Nothing. You feel the awful dread of impending doom creeping up on you, just like someone from Gen Z. On the far horizon - the second page of Google - you find this blog post shining at you like a sunrise.

Here is how you do it:

Setting up Mattermost webhook#

In the top left corner of your Mattermost, you can find a menu button. Click it. In the menu you find an option called Integrations.

Here you want to go to Create incoming webhook. This will create the endpoint that Grafana will use. Select which channel you want the bot to write its messages in.

In the form, there are two very important options, which should be filled like this:

This is brings the fierce dinosaur Grot the Grafanabot to life!

When you are done with the settings, save it and copy the URL that has been generated for you.

Setting up Grafana contact point#

In Grafana you want to configure a contact point. In the menu to the left hover the bell icon (Alerting) and select contact points. Here you want to click "New contact point".

Here comes the sneaky part

Under "Contact point type" select "Slack". 🙀 We fooled the system people!

Give the contact point a mandatory name and paste the URL you copied from the Mattermost webhook into "Webhook URL". You can configure the rest if you want, but it is not necessary and will override the settings you have made in Mattermost.

You can now freely use your contact point under "Notification policies" and Grot the Grafanabot will pop up in a channel near you with some news!

And granny will still get her email, of course.

/Tobias