SCOM Archives | adatum

Cookdown for SCOM monitor, extend and integrate

08/04/201906/04/2019
by Martin Ehrnst

It’s been a while since I worked daily with SCOM. But I still get my hands dirty with my old friend from time to time. For many years I used most of my time extending SCOMs functionality and integrating with other enterprise systems. I created a REST API before the SCOM had this available, and I have also created alot of custom management packs with PowerShell script monitors.
SCOM is one of the most used enterprise monitoring systems around, and companies will rely on it for many years to come. Integrations with SCOM will still be a key for many organizations. Luckiliy, you got a friend.

Cookdown launch

Cookdown is a new initiative aiming to blow new life in to your existing investment in SCOM and deliver stuff like ServiceNow integration and Easy Tune to help you out with those pesky overrides.

The team behind Cookdown will host a launch webinar on April 10. And if you’re interested in integration and extensions for SCOM you should definitely attend.

Azure Monitor

Microsoft killed SCOM internally

11/03/201911/03/2019
by Martin Ehrnst

Microsoft no longer uses SCOM to monitor their own workloads. They have replaced their entire SCOM based monitoring stack with Azure Monitor. Allegedly reduced alert noise and administration overhead.

Even if I have moved from SCOM as my main responsibility, I am still very much involved in the whole monitoring and management scope. Over the last years we have heard alot of talk about Azure Monitor replacing SCOM, but that cooled off after a while, maybe until now?

Technology change or cultural change

Microsoft’s story on how they killed SCOM internally was released one day before the official announcement on Operations Manager 2019. But we first heard the story at Ignite in 2018. One may ask, why the re-initiate this topic now?
For SCOM 2019, the focus is to better support hybrid cloud environments, which is good. If Microsoft doesen’t want to use it, should you?

I have written and spoken about the use of SCOM as your hub for Azure Monitor, and my opinion hasn’t changed that much. I belive that transition to you a new monitoring stack will happen with changes to the infrastructure.

When you read the article you’ll see that this was the case for Microsoft as well. There are two quotes i find partculary interesting in the announcement.

“This is not just a technology change, but a culture change,” Baxter says. “It wasn’t only that we would remove SCOM central monitoring, but we had to tell our application teams, now you’re going to manage alerts..”

It was January of 2017 when Baxter got the call. “Our goal was not just to get rid of SCOM, but to move to a Software as a Service (SaaS) solution and retire Virtual Machine (VM) based infrastructure,” she says.

The key here is change in culture. Microsoft went full on DevOps for their internal IT, and by doing that technology will change, and your monitoring will follow.
Further, the showcase mention monitoring was desentralized, which is true. But ther’s another key part of this story. The monitoring team built an integration service between their monitoring stack (Azure Monitor, app insights) and their ITSM system. This system allows for more meta data on each alert etc before ending up as a ticket.

Final notes

If you’re organization runs most of your IaaS on premises, you don’t have to make change yet. Allow the culture to drive the change. A long the way, your SCOM environment can be that integration service between Azure PaaS, FaaS, XaaS and ITSM.

Community

Speaking at SCOM Dagen 2018

01/10/201801/10/2018
by Martin Ehrnst

I am speaking on SCOM dagen 2018 which this year focuses on multi cloud, on premises and hybrid monitoring using Azure and SCOM.

The event focuses on how organisations can leverage new technology, and use their existing systems to become a better IT organisation. SCOM day is hosted by approved.se

My talk will scratch the surface on custom management pack development. Custom MP developement is used by service providers and larger organizations to gain better visibility in multi and hybrid cloud scenarios.

The event will also feature much more seasoned presenters like Thomas Maurer and Marcel Zehner

Hopefully I see you at SCOM dagen 2018 in Gothenburg

Windows Admin Center

Windows Admin Center with SquaredUp/SCOM

03/05/201803/05/2018
by Martin Ehrnst

Windows Admin Center Extension manager

Windows Admin Center (WAC) (formerly Project Honolulu) is the new, modern tool for managing servers, Hyper-V, clusters and even Windows 10 clients. Other than being fully HTML 5 driven it can integrate with third-party software via an Extension manager. All components within WAC is an extension, this means you can install, update, and delete individual components without re-installing the whole application.

As announced WAC SDK is now in public preview, meaning you and everyone else can build integrations with WAC. Luckily I have tested one integration already, and I have to say that the future looks promising.

Windows Admin Center integration with SCOM

I’ve been fortunate to be able to work closely with the team over at SquaredUp for many years now. At some point early 2018 they reached out and asked if I had looked in to Microsoft’s new server management tool, project Honolulu. SquaredUp told me they where working closely with the Windows Server team to make an integration with SCOM/SquaredUp and Honolulu.

Working for a large service provider in Norway we have many tools to provide management for our clients and a big priority is to integrate them to provide better visibility. I was immediately interested to integrate SCOM with Windows Admin Center. After installing the extension through extension manager you can access monitoring and historic performance data directly within server manager.

Top 3 for integrating SCOM and Windows Admin Center

Alerts within Microsoft’s latest management tool
Historic performance data combined with live telemetry
Maintenance mode directly from server manager

This is another good reason to drop interactive logons and get Windows Admin Center and I expect more to come over the next couple of days. If rumors are true we have some cool extensions showcased in this Microsoft Build 2018 session.

PS:

One way integration is cool, but I don’t see any reason for not to integrate WAC in other applications. If you feel the same way, please add your vote over at UserVoice to make that happen

Operations Manager

SCOM Virtualization host CPU spikes

14/03/201806/04/2018
by Martin Ehrnst

A lot of the core functionality SCOM 2016 has today was released with SCOM 2007. SCOM 2007 was released (as the name states) in 2007, at the very, very early stages of virtualization. 2007 Was also the start of my professional IT career and I remember only the most assertive companies with most capital was thinking about or using SAN and virtualization. I am talking about oil companies, large architectural firms etc. but still they had the environments in-house, making the virtualization environments small.

In 2018 most companies have much larger environments in-house or have moved everything to a service provider or a public cloud, and now, old SCOM 2007 implementations beginning to play a part.

Virtualization hosts

I work for a service provider in Norway, and we have around 4000 vm’s running on VMWare ESX. The environment is monitored in different ways, but visualization is using Grafana and Influx DB – providing very good insight to analyze the environment. See how you can create your own solution following Rudi Martinsens blog series on VMWare performance data.

This chart shows around 3000 VM’s CPU Ready spike every 15 minutes. Previously we had these spikes at 5 and 15. More on that later.

Collect Distributed Workflow Test Event

Collect Distributed Workflow Test Event is the rule that logs event id 6022 on all agent managed computers. It is used to “test event collection”.

Here’s a quote from the rule’s KB

This rule runs for each System Center Management Health Service and logs an event. This event is collected and used to verify that the end-to-end workflow to collect events properly is functioning as expected. If you alter the interval for this rule, it can cause the corresponding monitors to change state or generate an alert. The corresponding monitors are “No End to End Event for 45 Minutes (Critical Level)” and “No End to End Event for 30 Minutes (Warning Level)

The rule refers to two monitors using this event to check that “end-to-end” workflow is working. By default these two monitors are disabled, so what is the purpose of this rule? I already know from investigation that this rule indeed causes the CPU spikes every 15 minutes, that it has not implemented “spread initialization” which would be the prefered method. Instead it has a sync time forcing the same start interval for all agents. Even though it doesn’t create a noticeable overhead it self, multiply by X VMs on a host and you will see the impact.

I was not sure if the event logged by the rule was used to something else, so I reached out to Microsoft Premier Support. After a few phone calls and emails referring to my uservoice idea explaining the issue we got the following reply.

[…]
To summarize, if you did not enable the two monitors and if you have disabled the collection rule, logging the event is quite useless. There is no point in logging an event that no one checks afterwards. From this perspective, you could disable the rule logging the event and the collection rule as well, if this is not already disabled.

That confirmed my suspicions. This rule has no value (to our environment) and I can disable the whole thing.

Collect agent processor utilization

I have written about this rule exactly a year ago and I was not the first. It is the worst of the two and runs a script every five minutes to collect agent performance data. If you don’t use this data. Disable the rule.

Fun fact: Kevin Holman was the one suggested to run this rule every 321 seconds as he was tired of every workflow was running every 300 seconds by default.

Summary

Every SCOM environment differs from the other, but I strongly belive you are impacted by these two rules. “Collect Distributed Workflow Test Event” and “Collect agent processor utilization” both run on a fixed interval with a sync time instead of using Spread Initialization.

Depending on the size of your environment, , but if you don’t use the data generated by these rules I recommend you disable them. Here is a graph showing our two largest clusters hosting around 1000 VM’s.
Just before 11 I disabled “Collect Distributed Workflow Test Event” and you can clearly see the difference.

Let me know if you have experienced similar issues or have comments to this post.

Cookdown for SCOM monitor, extend and integrate

Cookdown launch

Share this:

Microsoft killed SCOM internally

Technology change or cultural change

Final notes

Share this:

Speaking at SCOM Dagen 2018

Share this:

Windows Admin Center with SquaredUp/SCOM

Windows Admin Center Extension manager

Windows Admin Center integration with SCOM

Share this:

SCOM Virtualization host CPU spikes

Virtualization hosts

Collect Distributed Workflow Test Event

Collect agent processor utilization

Summary

Share this: