Automation
SCOM Alerts to Microsoft Teams and Mattermost
…or slack?
For a SCOM 2016 implementation I have worked around different methods to consume alerts. Keeping emails to an absolute minimum and add more smartness to alerts and incidents is one of the main goals.
On a daily basis we use custom dashboards created with SquaredUp and integrated this with our ticketing and CMDB systems, allowing us to create alert tickets directly from SCOM on the connected to the correct server/person/customer etc. (Sounds like a great blog post down the road)
On the concept side of things, it shouldnt matter what system you use to handle alerts as long as someone takes action on it. So in this blog post I will show how you can ‘interact’ or at least notify on alerts with two collaboration tools. Microsoft Teams and Mattermost.
This example uses SCOM as the alert source, but it could easially be another monitoring system, Solarwinds, OMS, Datadog etc.
The main technique involves Webhooks which i used when sending alerts to Azure Automation, and is somewhat the same thing we are doing here.
On a high level, this is what we are going through.
- Creating Channels in MSFT Teams and Mattermost
- enabling them to receive incoming webhooks
- Create a new SCOM command channelAdd the PS script to send alerts
In teams, create a new channel by clicking the three bullets.( If you want to use an existing channel feel free).
After naming your channel, create a webhook for it by adding a new connector, and configure it.
Set a name and maybe an image, and remember to copy your URL.
In SCOM create a new command channel
You will ned the full path to powershell as the command file, which is:
C:\Windows\System32\WindowsPowershell\v1.0\Powershell.exe
And the startup folder
C:\Windows\System32\WindowsPowershell\v1.0\
The command line parameters are basicly a powershell script writtes as a ‘one-liner’ after the command parameter.
I will break it down for you here.
-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/webhook/************'
We are setting the execution policy and starting powershell with a command. The first “section” is a post to your webhook url.
Second, as the body i create a hash-table which holds our data and convert it to Json. For readability I have stripped down the code a little. We will insert data from SCOM here in the end.
-Body (ConvertTo-Json -InputObject @{'Title'='The title';'Text'='[View alert in SCOM Web console](https://scom)'}) -ErrorAction Stop}"
After you have created the command channel. Continue to add a new subscriber and a new subscription. When an alert matching your criteria is triggered you will se the following in your teams channel.
Success!
Mattermost: Using integrations from the menu. Create a new webhook assigned to the channel you want. Copy the webhook URL to use in your script, which for mattermost looks like this (should be somewhat equal to Slack). Not much changed from Teams.
-Body (ConvertTo-Json -InputObject @{'text'='alertname [View alert in SCOM](https://yoururl)';'username'='SCOM Alerts'}) -ContentType application/json -ErrorAction Stop}"
Please note that everything goes under ‘text’ and I have added a content type in our request as well an override for the poster username.
If all went well you should see a result like this:
Here are the two commands for each channel with SCOM data
Teams:
-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/*****' -Body (ConvertTo-Json -InputObject @{'Title'='$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ : $Data[Default='Not Present']/Context/DataItem/AlertName
Mattermost:
-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://mattermost/hooks/*******' -Body (ConvertTo-Json -InputObject @{'text'='$Data[Default='Not Present']/Context/DataItem/AlertName$ : $Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ [View alert in SCOM](https://URLid=$Data/Context/DataItem/AlertId$)';'username'='SCOM Alerts'}) -ContentType application/json -ErrorAction Stop}"
Notes:
As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.
The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.
To do this i will move the code out of scom and probably trigger an external script that does the logic. It could run serverless in azure functions, AWS Lambda, in azure automation, SMA or what ever you chose. I am looking to accomplish something like this and will of course let you know when its done.
from Slack documentation
References
https://msdn.microsoft.com/en-us/microsoft-teams/connectors
https://docs.mattermost.com/developer/webhooks-incoming.html
;'Text'='[View alert in SCOM Web console](https://URL/$Data/Context/DataItem/AlertId$)'}) -ErrorAction Stop}"
Mattermost:
Notes:
As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.
The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.
To do this i will move the code out of scom and probably trigger an external script that does the logic. It could run serverless in azure functions, AWS Lambda, in azure automation, SMA or what ever you chose. I am looking to accomplish something like this and will of course let you know when its done.
from Slack documentation
References
https://msdn.microsoft.com/en-us/microsoft-teams/connectors
https://docs.mattermost.com/developer/webhooks-incoming.html
