adatum

Getting your colleagues engaged with SCOM

08/02/201707/01/2025
by Martin Ehrnst

Q: How do I get my teammates engaged in SCOM?

A: Disable Email alerting

Q: What?

A: Let me explain what we did.

I had an interesting conversation the other day on how to get people more involved with SCOM. After going through what we have done over the years I figured it’s a topic worth sharing. Even if Operations Manager will be replaced with OMS or another product in the future, it’s still plays a significant role and will be for many more years.

In this post I will go through some of the steps I think have ben a deal breaker in terms of the involvement from application and service owners that have their environment monitored with SCOM. I won’t reproduce all the steps we have done as that will be to environment specific, but I will explain how and why i think you should look in to these topics.

Season SCOM with data from external sources.

By using data from your CMDB as additional properties to servers (and other objects) in SCOM makes a huge difference. I manage a large environment monitoring a lot of different customers with many services and applications running. Knowing which customer this server belongs to, what kind of backup it is running, which patch regime etc. This is data you typically find in a CMDB and by default SCOM is totally unaware of this.

We use a in-house developed CMDB system fully detached from any SCOM environment, but it has an API. By creating a management pack that extends the Windows computer class we now have the following extended properties on all servers monitored by SCOM

Customer name
Location
Type (Physical, virtual)
Host
Environment (Test, dev, production)
Services (Applications running)

All these properties can be used for almost anything. Group creation etc.

If you haven’t done this already, i strongly recommend connecting with a CMDB.

Replace the existing console.

The only person(s) who needs the SCOM console is the SCOM admin, and the only reasonable solution is to invest in a web-based system. Third party or in-house developed. There are a few commercial products out there, like SquaredUp and Savision, I encourage you to check them out. Below are two screenshots showing the difference between SquaredUp and the local console which should be a reason alone to invest in this.

SCOM Object state dashboard (who uses this?)

SquaredUp default installation showing a windows server object.

Agent tasks

This is a hidden gem. SCOM has an agent running on “all” servers in your environment, and this agent can run scripts for you by a click of a button. We have developed a management pack with a few tasks that was requested by my fellow colleagues.

Spend less time logging on to these servers and have the output directly.

Below is an output from the task showing disk free space. It is a simple Powershell script packed in a task targeted windows computers.

A few examples on other tasks

Add or remove management group
List local administrators
top x memory consuming processes
Restart agent
Start Windows service

Alert to ticket creation.

If youre not using SCSM, you probably havent got a good connection with your ticketing system or any at all. You can send an Email directly but chances are that it won’t work wery good. Let’s say you have an alert storm and you are sending alerts through a SMTP channel to your ticketing system. You will probably have 100 tickets created without any connection at all to the actual alert. Maybe you have two tickets for each alerts as well, one being resolution state NEW and the other Closed? Thats 200 tickets, or 198 because there are two business critical alerts not resolved but you don’t see it.

With SquaredUp we created a function for ‘on demand’ alert creation directly from the console using their built in functionality and a external script.

In a scenario with an alert storm the operations team can quickly look at their dashboards and see which alerts is still present – not the ones that are already resolved. Creating tickets for these alerts makes sense as they will have to be looked in to further. Below is a diagram showing how we set this up.

flowchart showing ticket creation from scom/squaredup

Along side with this flow. We update each ticket with a new message when the alert is closed.

Support different alert platforms.

What I mean is that you should try to integrate SCOM so that alerts can be consumed on other platforms. I have blogged earlier on how to post messages to Microsoft Teams and Mattermost. This can also be done with Slack. If you don’t use any of these collaboration tools, think and consult with your colleagues, they Probably have some great ideas!

Stop being personally involved with SCOM alerts.

As a SCOM administrator, how many times have you found your self invastigating an alert not within your field and without notifying anyone else? Probably too many. You’re not going to solve all the alerts and there’s is a reason for the application being monitored in the first place, someone wanted it. Sit down with your team and figure out a solution together.

Big Data and Events

Splunk. OMS, Elasticsearch. It doesn’t matter. If you manage to tie your existing SCOM environment with event and big data systems you will be amazed. Again, the built-in OMS, Event log and Web API plugins in SquaredUp can be used for this.

examples

Display SQL recommendations from the OMS SQL Assessment on all servers running SQL
Show change tracking events on the alert page.
Missing security updates on Windows server perspective.

Disable email alerts

It may be a bold statement, but if you manage to implement a few of the things i have listed and maintain your good tuning and MP implementation procedure. Chances are that you can start to disable alerting by email or at least get your colleagues more involved when they have the chance to properly use all the data and possibilities when having a SCOM installation in your environment.

Automation

Playing with cognitive services

16/01/201707/01/2025
by Martin Ehrnst

If you know what your users do or talk about you will likely have advantages over your competitors, or if you have support desk, you want to dispatch the ticket to the correct department as quickly as possible. To gain some insights you can use different AI/Machine learning tools to help you and ‘automatically’ perform actions.

Microsoft cognitive services is a set of APIs which let you do things like text analytics. I have played around a bit and found that I could pretty easy do a sentiment test (how ‘happy’ is the author) and a key phrase analysis (what is the text about). To do an analysis I needed to send the text in English. By living in Norway I am fortunate i many ways, but one of them is that Norwegian translate very good programmatically in to English

Since Microsoft (or Google and AWS) let’s us translate text through their translation API, you can in theory run text analysis on any language. I played around a bit and i managed to send some text through translation and in the end output a sentiment analysis and the key phrases. I set up the script in Azure Functions as well and it works pretty good.
To use it you will have to sign up for two Cognitive Services accounts in Azure, One for the Text Analytics API and one for the Translator API. In your Azure function you will have to set up the two API keys as variables.

The script is available on Github and it is totally a proof of concept without any error handling other than the APIs itself. Feel free to contribute to the code. Version when writing 0.5b

Here is an example on a text i found on a French news site.

Donald Trump a réaffirmé, lundi, ses positions critiques vis-à-vis de l’Otan, de l’UE, et de la politique d’accueil des migrants lors d’entretiens accordés à des médias européens. Une vision toujours proche de celle de Vladimir Poutine.

Une erreur catastrophique de Merkel sur l’accueil des migrants, l’Otan obsolète, le succès du Brexit qui marque le début de la fin de l’Union européenne. Si le fond ressemble à du Vladimir Poutine, la forme, elle, est clairement signée Donald Trump.

Lundi 16 janvier, à cinq jours de son investiture, le magnat de l’immobilier n’a pas mâché ses mots pour exposer ses vues sur les sujets d’actualité les plus brûlants sur le Vieux Continent, auprès des journaux britannique Times et allemand Bild.

Translated in to English

Donald Trump has r affirm, Monday, his criticism-screws – live NATO, the EU, and the migrant policy in interviews granted to European media. A vision still close to that of Vladimir Poutine.

A catastrophic error of Merkel on the reception of migrants, NATO MP4 you, the success of the Brexit brand the d to the end of the European Union. If the background looks like from Vladimir P
utin, the form, she is clearly sign e Donald Trump.

Monday, January 16, five days of his inauguration, the real estate mogul has no m ch her words to present its views on the topics of news the most br callers on the old Continent, aupr s of B
ritish newspapers Times and German Bild.

Not the best translation, but the analisys is quite OK

Sentiment Score : 87.73 %

Key phrases : Monday, Vladimir Putin, NATO MP4, Vladimir Poutine, criticism-screws - live NATO, aupr s of British newspapers Times, real estate mogul, end, European media, reception of migrants, m ch, old 
Continent, success, e Donald Trump, European Union, br callers, migrant policy, form, interviews, days, Brexit brand, inauguration, words, catastrophic error of Merkel, topics of news, German
Bild, background, January, vision

The tests done in Norwegian is pretty much spot on, and English analysis is just as you would expect.

Automation

SCOM Alerts to Microsoft Teams and Mattermost

06/01/201707/01/2025
by Martin Ehrnst

…or slack?

For a SCOM 2016 implementation I have worked around different methods to consume alerts. Keeping emails to an absolute minimum and add more smartness to alerts and incidents is one of the main goals.

On a daily basis we use custom dashboards created with SquaredUp and integrated this with our ticketing and CMDB systems, allowing us to create alert tickets directly from SCOM on the connected to the correct server/person/customer etc. (Sounds like a great blog post down the road)

On the concept side of things, it shouldnt matter what system you use to handle alerts as long as someone takes action on it. So in this blog post I will show how you can ‘interact’ or at least notify on alerts with two collaboration tools. Microsoft Teams and Mattermost.

This example uses SCOM as the alert source, but it could easially be another monitoring system, Solarwinds, OMS, Datadog etc.

The main technique involves Webhooks which i used when sending alerts to Azure Automation, and is somewhat the same thing we are doing here.

On a high level, this is what we are going through.

Creating Channels in MSFT Teams and Mattermost
enabling them to receive incoming webhooks
Create a new SCOM command channelAdd the PS script to send alerts

In teams, create a new channel by clicking the three bullets.( If you want to use an existing channel feel free).

After naming your channel, create a webhook for it by adding a new connector, and configure it.

Set a name and maybe an image, and remember to copy your URL.

In SCOM create a new command channel

You will ned the full path to powershell as the command file, which is:

C:\Windows\System32\WindowsPowershell\v1.0\Powershell.exe

And the startup folder

C:\Windows\System32\WindowsPowershell\v1.0\

The command line parameters are basicly a powershell script writtes as a ‘one-liner’ after the command parameter.

I will break it down for you here.

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/webhook/************'

We are setting the execution policy and starting powershell with a command. The first “section” is a post to your webhook url.

Second, as the body i create a hash-table which holds our data and convert it to Json. For readability I have stripped down the code a little. We will insert data from SCOM here in the end.

-Body (ConvertTo-Json -InputObject @{'Title'='The title';'Text'='[View alert in SCOM Web console](https://scom)'}) -ErrorAction Stop}"

After you have created the command channel. Continue to add a new subscriber and a new subscription. When an alert matching your criteria is triggered you will se the following in your teams channel.

Success!

Mattermost: Using integrations from the menu. Create a new webhook assigned to the channel you want. Copy the webhook URL to use in your script, which for mattermost looks like this (should be somewhat equal to Slack). Not much changed from Teams.

 -Body (ConvertTo-Json -InputObject @{'text'='alertname [View alert in SCOM](https://yoururl)';'username'='SCOM Alerts'}) -ContentType application/json  -ErrorAction Stop}"

Please note that everything goes under ‘text’ and I have added a content type in our request as well an override for the poster username.

If all went well you should see a result like this:

Here are the two commands for each channel with SCOM data

Teams:

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/*****' -Body (ConvertTo-Json -InputObject @{'Title'='$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ : $Data[Default='Not Present']/Context/DataItem/AlertName Mattermost:

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://mattermost/hooks/*******' -Body (ConvertTo-Json -InputObject @{'text'='$Data[Default='Not Present']/Context/DataItem/AlertName$ : $Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ [View alert in SCOM](https://URLid=$Data/Context/DataItem/AlertId$)';'username'='SCOM Alerts'}) -ContentType application/json  -ErrorAction Stop}"

Notes:

As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.

The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.

To do this i will move the code out of scom and probably trigger an external script that does the logic. It could run serverless in azure functions, AWS Lambda, in azure automation, SMA or what ever you chose. I am looking to accomplish something like this and will of course let you know when its done.

from Slack documentation

References

https://msdn.microsoft.com/en-us/microsoft-teams/connectors

https://docs.mattermost.com/developer/webhooks-incoming.html

;'Text'='[View alert in SCOM Web console](https://URL/$Data/Context/DataItem/AlertId$)'}) -ErrorAction Stop}"

Mattermost:

Notes:

As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.

The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.

from Slack documentation

References

https://msdn.microsoft.com/en-us/microsoft-teams/connectors

https://docs.mattermost.com/developer/webhooks-incoming.html

Getting your colleagues engaged with SCOM

Season SCOM with data from external sources.

Replace the existing console.

SCOM Object state dashboard (who uses this?)

SquaredUp default installation showing a windows server object.

Agent tasks

Alert to ticket creation.

Support different alert platforms.

Stop being personally involved with SCOM alerts.

Big Data and Events

Disable email alerts

Share this:

Playing with cognitive services

Share this:

SCOM Alerts to Microsoft Teams and Mattermost

from Slack documentation

from Slack documentation

Share this: