Azure Monitor Archives

Azure Lighthouse why is it so important

15/08/201920/12/2019
by Martin Ehrnst

Working for a Managed Service Provider (MSP) I have many times faced the challenges of managing multiple separate customers from one single pane. Whether it is a multi-tenant active directory, single AD or a vanilla Azure tenant. An MSP is only good when they can build tools to manage all customers in a streamlined fashion.

In the Microsoft sphere, partners and large enterprises have faced many of the same challenges. If you are a large enterprise, you might be eligible for an Enterprise Agreement.
As a partner, you can apply to become a (tier 1) Cloud Solution Provider (CSP). The tools provided are are far from good enough. The challenge is that you are still bound to tenant isolation. If you wanted to have a view of all alerts in Azure Monitor for all your customers. You need to create a tool that authenticates against each individual tenant and retrieves this information. Similar to what I did with SCOM.

Project Towboat

Last year I attended a side meeting for MSPs at Ignite. We discussed at scale management in the Azure Portal. We were promised that something called Project Towboat was planned. Since then it have been dead silent.
Out of the blue, Microsoft announced Azure Lighthouse. Promising simplified cross tenant resource management. So what makes this so great?

Delegated resource access

Azure Lighthouse uses delegated resource access. In essence, the customer establishes a trust with your (management/master) tenant. This allows for the users in the management directory (tenant) to manage resources on behalf of their customers. Many use Azure AD B2b to manage resources across multiple tenants. With Azure Lighthouse, you can do that without changing the context of the user.

In my opinion. Here are some of the features that make Azure Lighthouse so important to MSPs, and others managing multiple tenants.

Cross tenant monitoring in Azure

Azure Monitor is now multi-tenant. As long as the resource group or subscription is available for the person using Azure Monitor. Application and infrastructure monitoring is available from a single pane of glass.

Multi-tenant Log Analytics queries

Log Analytics is a part of Azure Monitor and is called Azure Monitor Logs, the engine behind is Log Analytics.
Log Analytics is already capable of searching within multiple workspaces. Since Azure Lighthouse will surface your customer’s workspaces, you can run cross tenant queries, how cool is that?

Azure security center for all customers

The beauty with delegated resource management just continues. Another great thing for your security team apart from Log Analytics is Azure Security Center is available in Azure Lighthouse. This means that the team (or that one person) can look at one single dashboard, or write the integration against one tenant.

Summary

With Azure Lighthouse greatly simplifies at scale and cross tenant management. Being tightly integrated with Azure Resource Manager for deployment, as well as Azure Monitor and Security Center for monitoring infrastructure and security.

I am really looking forward to creating solutions and working more with Azure Lighthouse. It is a long-awaited product, and with this launch, Microsoft is way ahead of its competitors.
Expect more dedicated posts on how to manage and automate using lighthouse in the future.

You can read more and find examples on the official Azure Lighthouse documentation and Azure Lighthouse github examples

Azure Monitor

Metric alerts for Azure monitor logs

12/06/201912/06/2019
by Martin Ehrnst

A common thing for traditional companies is to have one team responsible for monitoring. A few years ago, this team where close friends with the team provisioning infrastructure. Now, more and more companies are shifting to the “DevOps” world. Even Microsoft have killed SCOM and are only using Azure Monitor. Meaning that the one deployed the code (and the infrastructure) should be responsible for monitoring. In essence, this is great. But this transition takes time, and one should not underestimate the knowledge of the team who have been responsible for monitoring your entire infrastructure for decades.

If you are familiar with SCOM, you know that rules and monitors is targeted against a class of objects. IE, Windows 2016 operating system. When we move our workloads to Azure, we want to use Azure Monitor to monitor our workloads and VMs.

Enter Log alerts

Log Alerts has been around for quite some time and is commonly used to alert on actual log data. IE custom application logs, Windows event log and so on. But Log alerts has a “hidden” feature, especially for your monitoring teams, not wanting to manage hundreds of duplicate rules.

By using Log alerts with metric measurements you can almost replicate the what discoveries in SCOM does- find resources of a specific type, and attach some kind of monitoring to them. For example, you can create a search query for all your IaaS VMs and alert on their CPU counter.

This will let your monitoring team recreate all their logic, and have control over the entire infrastructure, almost as they had on-permises. At the same time you can leverage more DevOps practices and at the end have every team responsible for their own work.

Kusto examlpe

Below is a simple example that will list all VMs and their processor time. You can create an alert straight from Azure Monitor logs (former Log Analytics) or start from a new alert.

 Perf | where ObjectName == "Processor" and CounterName == "% Processor Time" | summarize AggregatedValue = avg(CounterValue) by bin(TimeGenerated, 5m), Computer

Summary

You have the option to monitor multiple VMs using one Alert Rule in Azure Monitor already. But one limitation is that this solution will not add new VMs to the alert rule. And for the time being, it only supports virtual machines
Log alerts are dependent on your query. So as long as your data is available, you can alert on it. Whether it is a web app, a SQL server or a custom log.

With Log Alerts, the transition to a public cloud-based infrastructure might be easier. Your operations teams can use their knowledge and re-create their on-premises monitoring logic as searches.
Application alerts could still be handled by the developers, and you can provision those using ARM templates or similar.

PS: I was going to write a longer post on how to manage and programmatically create log alerts, but with these great examples in Microsoft docs, there’s no need to re-invent the wheel.

Azure Monitor

Microsoft killed SCOM internally

11/03/201911/03/2019
by Martin Ehrnst

Microsoft no longer uses SCOM to monitor their own workloads. They have replaced their entire SCOM based monitoring stack with Azure Monitor. Allegedly reduced alert noise and administration overhead.

Even if I have moved from SCOM as my main responsibility, I am still very much involved in the whole monitoring and management scope. Over the last years we have heard alot of talk about Azure Monitor replacing SCOM, but that cooled off after a while, maybe until now?

Technology change or cultural change

Microsoft’s story on how they killed SCOM internally was released one day before the official announcement on Operations Manager 2019. But we first heard the story at Ignite in 2018. One may ask, why the re-initiate this topic now?
For SCOM 2019, the focus is to better support hybrid cloud environments, which is good. If Microsoft doesen’t want to use it, should you?

I have written and spoken about the use of SCOM as your hub for Azure Monitor, and my opinion hasn’t changed that much. I belive that transition to you a new monitoring stack will happen with changes to the infrastructure.

When you read the article you’ll see that this was the case for Microsoft as well. There are two quotes i find partculary interesting in the announcement.

“This is not just a technology change, but a culture change,” Baxter says. “It wasn’t only that we would remove SCOM central monitoring, but we had to tell our application teams, now you’re going to manage alerts..”

It was January of 2017 when Baxter got the call. “Our goal was not just to get rid of SCOM, but to move to a Software as a Service (SaaS) solution and retire Virtual Machine (VM) based infrastructure,” she says.

The key here is change in culture. Microsoft went full on DevOps for their internal IT, and by doing that technology will change, and your monitoring will follow.
Further, the showcase mention monitoring was desentralized, which is true. But ther’s another key part of this story. The monitoring team built an integration service between their monitoring stack (Azure Monitor, app insights) and their ITSM system. This system allows for more meta data on each alert etc before ending up as a ticket.

Final notes

If you’re organization runs most of your IaaS on premises, you don’t have to make change yet. Allow the culture to drive the change. A long the way, your SCOM environment can be that integration service between Azure PaaS, FaaS, XaaS and ITSM.

Azure

Working with Azure Monitor Rest API

14/11/201719/01/2020
by Martin Ehrnst

Before delving in to Azure monitor Rest API and powershell, let’s take a little step back. Azure monitor released in public preview a little over a year ago (September 2016). Introduced as “The built-in solution to make monitoring available for all Azure users”.
At that time I was personally all over the Operations Management Suite or OMS which is now a deprecated brand. All features from OMS is now available under “Monitoring + Management” in the Azure Marketplace.

Therefore, Azure Monitor went a bit under my radar, but when OMS shifted we started to see more and more about Azure Monitor being the one stop shop for Azure monitoring. Especially the alerting feature seems to be richer in Monitor than in Log Analytics and it is my (and others) anticipation that we will see Monitor as the default alert tool, for both metrics and activity logs.

So as part of my never-ending story, Monitoring Azure as a CSP provider. Let’s take a look at Azure Monitor and it’s REST API

We want to retrieve alert rules and incidents (alerts) programmatically, but first we create an alert rule to work with through the GUI. In th Azure portal:

At this time of writing the alert rule is bound to a specific resource. I hope we will see the ability to create rules based on resource type. ie: you want all web apps to have the same standard alert rules for response time.

Locate Azure monitor

Find your resource and metric (or you can jump straight in to alerts)

Verify your alert rule exist

Retrieve Azure Monitor alerts and incidents

From Powershell I am connecting to AAD and generating authentication header, reusing code from earlier blog post about Azure Resource health. For the purpose of this post I will focus on these two Azure Monitor API endpoints: Alert Rules – List By Resource Group and Alert Rule Incidents – List By Alert Rule.

After you have authenticated against Azure AD, and if using my previous sample you should have the following header available.

Using this header we call the alert rules endpoint to get our alert rules. Pay attention to the URL, as it requires you to specify a subscription id and the resource group name.

The output from the above should look something like this. I only have one alert configured for the resource group

id : /subscriptions/a2782f8e/resourceGroups/2017/providers/microsoft.insights/alertrules/name name : name type : Microsoft.Insights/alertRules location : westeurope tags : @{$type=Microsoft.WindowsAzure.Management.Common.Storage.CasePreservedDictionary,} properties : @{name=No runs; description=; isEnabled=True; condition=; action=; lastUpdatedTime=2017-11-02T12:52:26.9091865Z; provisioningState=Succeeded; actions=System.Object[]}

Next we use the ID from our previus result as part of the URL to get our alert incidents

I am jumping straight in to the ‘value’ node at this point. If youre alert rule have triggered an incident it will return a result. We see the time it was activated (and resolved if it’s old), a boolean value of its status, and some information on the resource it self.

id : L3N1YnNjcmlwdGlvbnMvZTUxNGRhY2EtYTA3Ny00NGYwLTljZmEtNjBlMzRjOTk1Zjk3L3Jlc291cmNlR3JvdXBzL1NlbWluYXIyMDE3L3Byb3ZpZGVycy9taWNyb3NvZnQuaW5zaWdodHMvYWxlcnRydWxlcy9ObyUyMHJ1bnMwNjM2NDUyMjM3NjcyODMwODI2 ruleName : /subscriptions/a2782f8e/resourceGroups/2017/providers/microsoft.insights/alertrules/name isActive : True activatedTime : 2017-11-02T12:49:27.2830826+00:00 resolvedTime : targetResourceId : targetResourceLocation : legacyResourceId :

Below I have tied everything together, using my Azure App authentication function we generate the auth header and retrieves alert rules based on user input.

Wrapping up

We now know the basic consepts on how we authenticate and retrieve alert rules and incidents from Azure Monitor through it’s rest API using Powershell. From here we can easially expand our work to create new alert rules or retrieve metrics from our resources wich lets us build custom solutions on prem or any where else.

If you want to continue exploring Azure Monitoring capabilities i suggest you follow Adin Ermies series describing all the different solutions.

Hopefully I can soon provide some insights on how we are building our CSP monitoring solution in a single blog post, using the different tools mentioned in my latest posts.

Azure Lighthouse why is it so important

Project Towboat

Delegated resource access

Cross tenant monitoring in Azure

Multi-tenant Log Analytics queries

Azure security center for all customers

Summary

Share this:

Metric alerts for Azure monitor logs

Enter Log alerts

Kusto examlpe

Summary

Share this:

Microsoft killed SCOM internally

Technology change or cultural change

Final notes

Share this:

Working with Azure Monitor Rest API

Retrieve Azure Monitor alerts and incidents

Wrapping up

Share this: