OpsMgr Archives - Page 7 of 12

Getting your colleagues engaged with SCOM

08/02/201707/01/2025
by Martin Ehrnst

Q: How do I get my teammates engaged in SCOM?

A: Disable Email alerting

Q: What?

A: Let me explain what we did.

I had an interesting conversation the other day on how to get people more involved with SCOM. After going through what we have done over the years I figured it’s a topic worth sharing. Even if Operations Manager will be replaced with OMS or another product in the future, it’s still plays a significant role and will be for many more years.

In this post I will go through some of the steps I think have ben a deal breaker in terms of the involvement from application and service owners that have their environment monitored with SCOM. I won’t reproduce all the steps we have done as that will be to environment specific, but I will explain how and why i think you should look in to these topics.

Season SCOM with data from external sources.

By using data from your CMDB as additional properties to servers (and other objects) in SCOM makes a huge difference. I manage a large environment monitoring a lot of different customers with many services and applications running. Knowing which customer this server belongs to, what kind of backup it is running, which patch regime etc. This is data you typically find in a CMDB and by default SCOM is totally unaware of this.

We use a in-house developed CMDB system fully detached from any SCOM environment, but it has an API. By creating a management pack that extends the Windows computer class we now have the following extended properties on all servers monitored by SCOM

Customer name
Location
Type (Physical, virtual)
Host
Environment (Test, dev, production)
Services (Applications running)

All these properties can be used for almost anything. Group creation etc.

If you haven’t done this already, i strongly recommend connecting with a CMDB.

Replace the existing console.

The only person(s) who needs the SCOM console is the SCOM admin, and the only reasonable solution is to invest in a web-based system. Third party or in-house developed. There are a few commercial products out there, like SquaredUp and Savision, I encourage you to check them out. Below are two screenshots showing the difference between SquaredUp and the local console which should be a reason alone to invest in this.

SCOM Object state dashboard (who uses this?)

SquaredUp default installation showing a windows server object.

Agent tasks

This is a hidden gem. SCOM has an agent running on “all” servers in your environment, and this agent can run scripts for you by a click of a button. We have developed a management pack with a few tasks that was requested by my fellow colleagues.

Spend less time logging on to these servers and have the output directly.

Below is an output from the task showing disk free space. It is a simple Powershell script packed in a task targeted windows computers.

A few examples on other tasks

Add or remove management group
List local administrators
top x memory consuming processes
Restart agent
Start Windows service

Alert to ticket creation.

If youre not using SCSM, you probably havent got a good connection with your ticketing system or any at all. You can send an Email directly but chances are that it won’t work wery good. Let’s say you have an alert storm and you are sending alerts through a SMTP channel to your ticketing system. You will probably have 100 tickets created without any connection at all to the actual alert. Maybe you have two tickets for each alerts as well, one being resolution state NEW and the other Closed? Thats 200 tickets, or 198 because there are two business critical alerts not resolved but you don’t see it.

With SquaredUp we created a function for ‘on demand’ alert creation directly from the console using their built in functionality and a external script.

In a scenario with an alert storm the operations team can quickly look at their dashboards and see which alerts is still present – not the ones that are already resolved. Creating tickets for these alerts makes sense as they will have to be looked in to further. Below is a diagram showing how we set this up.

flowchart showing ticket creation from scom/squaredup

Along side with this flow. We update each ticket with a new message when the alert is closed.

Support different alert platforms.

What I mean is that you should try to integrate SCOM so that alerts can be consumed on other platforms. I have blogged earlier on how to post messages to Microsoft Teams and Mattermost. This can also be done with Slack. If you don’t use any of these collaboration tools, think and consult with your colleagues, they Probably have some great ideas!

Stop being personally involved with SCOM alerts.

As a SCOM administrator, how many times have you found your self invastigating an alert not within your field and without notifying anyone else? Probably too many. You’re not going to solve all the alerts and there’s is a reason for the application being monitored in the first place, someone wanted it. Sit down with your team and figure out a solution together.

Big Data and Events

Splunk. OMS, Elasticsearch. It doesn’t matter. If you manage to tie your existing SCOM environment with event and big data systems you will be amazed. Again, the built-in OMS, Event log and Web API plugins in SquaredUp can be used for this.

examples

Display SQL recommendations from the OMS SQL Assessment on all servers running SQL
Show change tracking events on the alert page.
Missing security updates on Windows server perspective.

Disable email alerts

It may be a bold statement, but if you manage to implement a few of the things i have listed and maintain your good tuning and MP implementation procedure. Chances are that you can start to disable alerting by email or at least get your colleagues more involved when they have the chance to properly use all the data and possibilities when having a SCOM installation in your environment.

Automation

SCOM Alerts to Microsoft Teams and Mattermost

06/01/201707/01/2025
by Martin Ehrnst

…or slack?

For a SCOM 2016 implementation I have worked around different methods to consume alerts. Keeping emails to an absolute minimum and add more smartness to alerts and incidents is one of the main goals.

On a daily basis we use custom dashboards created with SquaredUp and integrated this with our ticketing and CMDB systems, allowing us to create alert tickets directly from SCOM on the connected to the correct server/person/customer etc. (Sounds like a great blog post down the road)

On the concept side of things, it shouldnt matter what system you use to handle alerts as long as someone takes action on it. So in this blog post I will show how you can ‘interact’ or at least notify on alerts with two collaboration tools. Microsoft Teams and Mattermost.

This example uses SCOM as the alert source, but it could easially be another monitoring system, Solarwinds, OMS, Datadog etc.

The main technique involves Webhooks which i used when sending alerts to Azure Automation, and is somewhat the same thing we are doing here.

On a high level, this is what we are going through.

Creating Channels in MSFT Teams and Mattermost
enabling them to receive incoming webhooks
Create a new SCOM command channelAdd the PS script to send alerts

In teams, create a new channel by clicking the three bullets.( If you want to use an existing channel feel free).

After naming your channel, create a webhook for it by adding a new connector, and configure it.

Set a name and maybe an image, and remember to copy your URL.

In SCOM create a new command channel

You will ned the full path to powershell as the command file, which is:

C:\Windows\System32\WindowsPowershell\v1.0\Powershell.exe

And the startup folder

C:\Windows\System32\WindowsPowershell\v1.0\

The command line parameters are basicly a powershell script writtes as a ‘one-liner’ after the command parameter.

I will break it down for you here.

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/webhook/************'

We are setting the execution policy and starting powershell with a command. The first “section” is a post to your webhook url.

Second, as the body i create a hash-table which holds our data and convert it to Json. For readability I have stripped down the code a little. We will insert data from SCOM here in the end.

-Body (ConvertTo-Json -InputObject @{'Title'='The title';'Text'='[View alert in SCOM Web console](https://scom)'}) -ErrorAction Stop}"

After you have created the command channel. Continue to add a new subscriber and a new subscription. When an alert matching your criteria is triggered you will se the following in your teams channel.

Success!

Mattermost: Using integrations from the menu. Create a new webhook assigned to the channel you want. Copy the webhook URL to use in your script, which for mattermost looks like this (should be somewhat equal to Slack). Not much changed from Teams.

 -Body (ConvertTo-Json -InputObject @{'text'='alertname [View alert in SCOM](https://yoururl)';'username'='SCOM Alerts'}) -ContentType application/json  -ErrorAction Stop}"

Please note that everything goes under ‘text’ and I have added a content type in our request as well an override for the poster username.

If all went well you should see a result like this:

Here are the two commands for each channel with SCOM data

Teams:

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://outlook.office365.com/*****' -Body (ConvertTo-Json -InputObject @{'Title'='$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ : $Data[Default='Not Present']/Context/DataItem/AlertName Mattermost:

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'https://mattermost/hooks/*******' -Body (ConvertTo-Json -InputObject @{'text'='$Data[Default='Not Present']/Context/DataItem/AlertName$ : $Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$ [View alert in SCOM](https://URLid=$Data/Context/DataItem/AlertId$)';'username'='SCOM Alerts'}) -ContentType application/json  -ErrorAction Stop}"

Notes:

As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.

The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.

To do this i will move the code out of scom and probably trigger an external script that does the logic. It could run serverless in azure functions, AWS Lambda, in azure automation, SMA or what ever you chose. I am looking to accomplish something like this and will of course let you know when its done.

from Slack documentation

References

https://msdn.microsoft.com/en-us/microsoft-teams/connectors

https://docs.mattermost.com/developer/webhooks-incoming.html

;'Text'='[View alert in SCOM Web console](https://URL/$Data/Context/DataItem/AlertId$)'}) -ErrorAction Stop}"

Mattermost:

Notes:

As you see I have put the scripts which handles the logic in a powershell onliner, inside a SCOM channel.

The next thin I want to do is to add more logic to the messages posted in Teams/Slack/Mattermost like taking actions etc.

from Slack documentation

References

https://msdn.microsoft.com/en-us/microsoft-teams/connectors

https://docs.mattermost.com/developer/webhooks-incoming.html

OMS

SCOM and OMS: The agents

01/12/201607/01/2025
by Martin Ehrnst

Both SCOM and OMS are in most ways dependent on agents installed on each server being monitored. If you have SCOM today you probably run Microsoft Operations Manager Agent (MOM) which is shipped with SCOM 2012 media and installed through the SCOM console or a SW deployment tool ConfigMgr etc.

OMS on the other hand uses the new Microsoft Monitoring Agent (MMA) which is available for download within your OMS workspace or through MSFT download center. MMA is said to be the mother of all agents which should be used by every Microsoft software that require an agent.

NB: At some point (SCOM 2012 SP1?) the agent actually got rebranded to Microsoft Monitoring Agent. It is stil not the same you get from OMS

The reason for looking in to the agents is that we one situation wher we lost contact with one server belonging to a customer (a customer of my employee Intility).

It turned out that they had signed up for OMS and configured the MMA agent to. At some point this deleted our SCOM configuration and we lost monitoring on that server.

As OMS is fast forwarding these days we will have to be proactive and support agent/servers directly reporting to other OMS workspaces than the one connected to our SCOM environment.

After some testing here is what i found is the main differences on these agents:

Supports	MOM/SCOM Agent	OMS Agent (MMA)
SCOM Workgroup	X	X
Multihome SCOM Workgroup	X	X*
OMS Reporting	X (through SCOM)	X
Multihome OMS Workspace		X*
Hybrid Worker support		X

*I do not know if there is a limitation on how many connections an agent can have, but I have tried 5 in total. 3 OMS workspaces and 2 SCOM work groups.

Screenshots showing MMA

MOM rebranded

At this point we plan to upgrade all agents making sure customers are able to report to their own OMS workspace without interfering with our central SCOM and OMS installations.

SCOM 2016 is also shipped with the MMA agent and is likely the same but with a different version.

Let me know if you have something to add and I will update my post accordingly.

Bombshell:

If you extract the .MSI from MMASetup******.exe you end up with a MSI which installs the “old” MOM agent

Getting your colleagues engaged with SCOM

Season SCOM with data from external sources.

Replace the existing console.

SCOM Object state dashboard (who uses this?)

SquaredUp default installation showing a windows server object.

Agent tasks

Alert to ticket creation.

Support different alert platforms.

Stop being personally involved with SCOM alerts.

Big Data and Events

Disable email alerts

Share this:

SCOM Alerts to Microsoft Teams and Mattermost

from Slack documentation

from Slack documentation

Share this:

SCOM and OMS: The agents

Share this: