CSP Monitoring Archives

Azure monitoring, connecting the dots

04/01/201822/01/2018
by Martin Ehrnst

Azure Monitoring

Welcome to the continuing saga on how to monitor your customers Azure tenants being a service provider. Previously we have covered how to authenticate against Microsoft CSP, using Azure Resource Health API with Powershell and more.

This post is all about connecting the dots. We are far away from finished, but things are moving in this project and at the time of writing, we have two separate projects going.
The first one is focused on creating a single pane of glass for all our customers’ workflows. This involves custom coding and management pack development for SCOM. The second one, which this post will cover, is how we have designed each customer tenant and how we plan to use built-in Azure monitoring functionality.

Customer tenant setup

Working for a service provider we need to construct Azure tenants by taking in to account that we are going to manage cloud resources, so using many cloud features makes a lot of sense. The challenge ist that we always have to think about how we can integrate with an existing deployment and work with monitoring solutions on premises.

When we first started out this project we looked in-to what have been done before, and most of the examples we found wouldn’t scale to our requirements or used OMS/Log Analytics only. We wanted to use our SCOM environment for alert handling, dashboard and platform health as SCOM is already integrated with customer portals, CMDBs and more. We will discuss more on that later in this blog post.

Things are moving very fast in Azure, we have changed our inital customer tenant setup twice before we found a structure we believe is future friendly.
When a customer sign up for an Azure Subscription, we populate their tenant with a default monitoring resource group and a OMS/Log Analytics workspace (LA). Along with the default LA workspace we add the Azure Activity Log, Web Apps and Office 365 solutions as standard.
For “bread and butter” type of Azure Resources, such as compute and web apps we setup the same type of monitoring regime we provide for on-premise resources, but we use alerts in Azure Monitor. This approach works well for Azure Resources which do not have existing, custom Log Analytics solutions and searches to provide health state. This means that VMs deployed using our custom ARM template will also include Monitor Alerts such as “CPU Usage % above 95” and “Web app response time above x”. In conjunction with Azure Monitor we use Azure Resource Health wich will provide health state data regardless of resource type, and custom alerts in monitor or Log Analytics.

Below is a (not so detailed) illustration on our default tenant.

SCOM and Azure Integration

We use System Center Operations Manager (SCOM) as our main monitoring platform for operating systems and applications. As SCOM is already integrated with our ticketing system, CMDB and other internal tools it seems reasonable to provide insight to application and workloads running in Azure on the same monitoring platform. That way we optimally can provide a single pane of glass in to the on premise, hybrid and cloud only workloads.

Azure Management Packs

To get monitoring data in to our on prem SCOM we looked in to two major options.

Option #1:
The official Azure Management pack from Microsoft. he official MP discovery process/adding new tenants cannot be automated. It relies on a GUI where you sign it to the tenant etc. neither does it provide any “umbrella” functionality for companies enrolled in the CSP program.

Option #2:
Daniele Grandini’s Azure/OMS management pack. Daniele’s management packs provide insight to Log Analytics, Azure Backup and Automation, but relies on the official Microsoft MP for initial discovery. Daniele’s management packs focuses on the solutions within the “Monitoring + management” (formerly known as OMS) space in Azure. Since much of the alerting features from OMS/Log Analytics are moving to Azure Monitor, I reached out to Daniele and asked if he had looked in to creating a management pack for that. He had looked a little in to it, but was also concerned about the rapid changes. Unfortunately this MP is bound to the initial discovery from the official Azure MP. A service provider managing several hundred tenants (and growing) cannot have that limitation. I hope to be able to help Daniele with the upcoming Azure Monitor MP.

Here’s where our problems started. I wanted to discover all our manged tenants automatically. Take advantage of being a CSP we set out to create our own management pack(s). I have create one management pack for the CSP platform that integrate with the Partner Center API (see example in this blog post) to do the initial discovery. Tenants and subscriptions are populated as objects in SCOM. Further, using a Partner Center Managed Application we can pre-consent access to all managed tenants. That means we can use this applications credentials to authenticate against each of our managed tenants, by-passing the limitation within the official management pack. All resources are the created as object with a hosting relationship to resourcegroup, subscription and tenant. Basic monitoring is done through Azure Resource Health API.

Below is a diagram showing the structure of our CSP management pack

Credentials used to authenticate against partner center and the Azure tenants is provided through SCOM RunAs accounts.

Our next step in SCOM and Azure integration is to create an Azure Monitor Management pack that reference the CSP management pack. This will provide the more enriched monitoring provided by Azure Monitor. Due to many recent changes to the monitor platform I have decided to wait and see where we end up. At the time of writing Azure Monitor have two new alert features in preview and none of their API’s are officially documented – i will come back with examples when I have something tangible.

Summary

To provide effective monitoring as a service provider for customers which span on-prem and cloud environments, we recommend the following:

For “bread and butter” monitoring use a combination of SCOM and Azure Monitor
If in the CSP program. Create a management pack using CSP rest API’s (hopefully I can share our MP later) combined with a custom Azure Monitor MP
Not a CSP? Look in to a combination of the official MP and Daniele’s management packs.
Deploy Log Analytics as default to all tenants. This will give you an advantage when customers require custom solutions and log sources.

Wrapping up

All service providers do their monitoring differently, but hopefully you have gotten some ideas on how you can do yours. Our solution is far from being finished, but I feel we have a structure that are future proof (the modern type of future). Hopefully we can share the SCOM management packs later, but feel free to contact me on specifics. Just remember I cannot share the MP itself at this point in time.

Until further notice, this will be the closing post on how you can do Azure Monitoring as a service provider.

Big thanks to Kevin Green and Cameron Fuller for providing feedback and to reach out to other community friends on my behalf.

Azure

Resource health through Azure Rest API

18/09/201725/11/2019
by Martin Ehrnst

As a part of a large monitoring project involving on prem, Azure and Office 365 I have started to explore the different methods on how to acquire all relevent data. Previously, I have written a post on how you authenticate against Microsoft’s partner center API which is a part of the same feasibility project.
Later down the road i will try to write a larger blog series on how we can monitor Microsoft Cloud and on-Prem resources. Hopefully it will be joyful…

This blog post will describe how you can use Powershell to authenticate and get availability status from all resource groups and their resources. If you’re not that interested in monitoring data, use this post as a guide on how to get started with the API and the rest is documented on the Azure API documentation pages.

High level overview:

Set up an Azure Active Directory Application to authenticate (not covered)
Build an authentication header with a token from Azure AD
Get all resource groups within a subscription
Get the availability of all resources within a resource group

Get Azure AD application token

After setting up/registering the application in Azure AD you will have to use the application ID and secret in order to generate an authentication token to use against Azure management Rest API’s. I have created a basic Powershell function you can use, including an example authentication header.

$result = Get-AADAppoAuthToken –ClientID <AzureAD APPLICATION ID> –ClientSecret <ClientSecret> –TenantId "test.no"
$AuthKey = "Bearer " + ($result.access_token)
$authHeader = @{
 'Content-Type' = 'application/json'
 'Accept' = 'application/json'
 'Authorization' = $AuthKey
}

view raw
createAuthHeaderExample.ps1
hosted with ❤ by GitHub

 <#
 .SYNOPSIS
 Function to connect to the Microsoft login OAuth endpoint and return an OAuth token.
 .DESCRIPTION
 Generate Azure AD oauth token.
 You can specify the resource you want in the paramenter. Default is management.core.windows.net
 Parts of this function is created from these examples: https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-rest-api-walkthrough
  .PARAMETER ClientID
 Azure AD application ID
  .PARAMETER ClientSecret
 Your application secret.
  .PARAMETER TenantId
 Your tenant domain name. test.onmicrosoft.com
  .PARAMETER ResourceName
 Specify if you are accessing other resources than https://management.core.windows.net
 For example microsoft partner center would have https://api.partnercenter.microsoft.com
 .EXAMPLE
 Get-AADAppoAuthToken -ClientID 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' -ClientSecret <application secret> -TenantId "test.no" will return
 token_type     : Bearer
 expires_in     : 3600
 ext_expires_in : 0
 expires_on     : 1505133623
 not_before     : 1505129723
 resource       : https://management.core.windows.net/
 access_token   : eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkhIQnlLVS0wRHFBcU1aaDZaRlBkMlZXYU90ZyIsImtpZCI6IkhIQnlLVS0wRHFBcU1aaDZaRlB
 kMlZXYU90ZyJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy
 
 .NOTES
 v1.0
 Martin Ehrnst 2017
 #>
 [Cmdletbinding()]
 Param(
 [Parameter(Mandatory = $true)]
 [string]$ClientID,
 [Parameter(Mandatory = $true)]
 [string]$ClientSecret,
 [Parameter(Mandatory = $true)]
 [string]$TenantId,
 [Parameter(Mandatory = $false)]
 [string]$ResourceName = "https://management.core.windows.net/"
 )
 $LoginURL = 'https://login.windows.net'
 #Get application access token
 $Body = @{
 grant_type = "client_credentials";
 resource = $ResourceName;
 client_id = $ClientID;
 client_secret = $ClientSecret
 }
 Return Invoke-RestMethod –Method Post –Uri $LoginURL/$TenantId/oauth2/token –Body $Body
}

view raw
Get-AADAppoAuthToken.ps1
hosted with ❤ by GitHub

Get all resource groups and resources

Next we will grab all resource groups within a subscription, before looping through each resource group to get the individual resources.

#Loop through each reasource group and get all resources.
#Add everything to a hash table
$Groups = @()
foreach ($rg in $ResourceGroups) {
 $ResourceGroupUri = "https://management.azure.com/subscriptions/$subscriptionID/resourceGroups/$rg/resources?api-version=$APIVersion"
 $res = (Invoke-RestMethod –Uri $ResourceGroupUri –Method GET –Headers $authHeader).value
 #Create array of all resources
 $resources = @{}
 $resources.Add($rg, $res)
 #Add all resource groups and their resources to a hash table
 $Groups += $resources
}

view raw
GetAllResorurcesWithinRG.ps1
hosted with ❤ by GitHub

#get all resource groups within a subscription
$APIVersion = "2017-05-10"
$subscriptionID = "xxxxxxxxxxxx-xxx-xxx-xxxxxxxxxx"
$RGURI = "https://management.azure.com/subscriptions/$subscriptionID/resourcegroups?api-version=$APIVersion"
$ResourceGroups = (Invoke-RestMethod –Uri $RGuri –Method GET –Headers $authHeader).value.name

view raw
GetAzureRG.ps1
hosted with ❤ by GitHub

Now that we have all resource group and all resources within we can use this to further get the resource health and availability. To be honest we don’t need to get RG before checking the resources it self, but as a starting point i find it very useful to have the resource groups and their resources available within my scripts.

Get the resource availability status

Now we have authenticated, grabbed all resource groups and their resources and it’s time to find our resource health. There are multiple ways on how to get this data, and we have to consider the methods we use. One limitation i hit pretty fast working with this is the number of subscription resource get requests, specified in the response header “x-ms-ratelimit-remaining-subscription-resource-requests: ” All limitations documentet, here

When you hit this limit every substantial requests is dropped. I have to rethink my whole monitoring scenario due to these limitations.

Get Resource health based on resource group

This let’s us get health state on all resources scoped to a resource group. If we reuse the data from previously (we have all resource groups within our subscription) we will get all resource group health this way.

availabilityState : Unavailable
summary           : Your virtual machine is unavailable
detailedStatus    : We're working to automatically recover your virtual machine and to determine the source of the problem. No additional action is required from you at this time.
reasonType        : Unplanned
occuredTime       : 2017-07-30T01:13:56Z
reasonChronicity  : Persistent
reportedTime      : 2017-09-12T11:27:42.3921293Z
resolutionETA     : 2017-07-30T01:38:56Z

view raw
example-output
hosted with ❤ by GitHub

#get the health of the whole resource group
# Add each health status to a hashtable before output a complete table with all resource groups and their resource health
$resourceGroupHealth = @{}
foreach ($ResourceGroup in $ResourceGroups) {
 #Set resource group name and use it in our url
 $health = Invoke-RestMethod –Uri "https://management.azure.com/subscriptions/$subscriptionID/resourceGroups/$ResourceGroup/Providers/Microsoft.ResourceHealth/availabilityStatuses?api-version=2015-01-01" –Method GET –Headers $authHeader
 $currentHealth = @{}
 $currentHealth = @{
 [string]"$ResourceGroup" = [object]$health
 }
 $resourceGroupHealth += $currentHealth
}
$resourceGroupHealth
#Explore the results
$resourceGroupHealth.item('ResourceGroup').Value.Properties

view raw
GetResourceGroupHealth.ps1
hosted with ❤ by GitHub

Other examples are to get availability by Resource and the entire subscription

Azure

Authenticate against Micrsoft Partner Center API using Powershell

07/09/201719/12/2019
by Martin Ehrnst

Update 04.01.2019:
While the method described in this post still work. Microsoft are moving to what they call secure app model. Meaning that password grant is deprecated and you will need to use a refresh token model. I have written a new blog post, explaining the new model.

If you’re not familiar with the Microsoft Cloud Service Provider program it’s in short a program to easier let service providers manage their customers tenants and subscriptions within Azure and Office 365 from a centralized platform.

Apart from a very limited web portal it have a set of API’s and SDK’s to build your own solutions – wich I assume is prefered from Microsoft and the service provider. For a project I needed to authenticate against the REST API using Powershell and then retrieve some information about each tenant, who would have thought that could be so much work

Here’s what I said.

That’s fine, I will have it to you in an hour.

For your reference, this is the API I am working with: Partner Center Swagger

An hour later I did have authentication in place, but I was unable to retrieve any information from our customers. After digging through the documentation I found that the customer endpoints required “App + User Authentication” where I had only authenticated with AppId and App Secret.

After spending too much time decifer the C# examples on how you authenticate with app and user against the CSP Rest API i finally had a working Powershell function.

These are the steps required

Generate a token from Azure AD by calling https://login.microsoft.com/tenant-name/oauth/token
- Specified with the resource you want to access (partner center api), client id, username and password, correct grant type and scope
Use the AAD token to authenticate against partnercenter/generatetoken and recieve a correct User + App jwt_token
Use the jwt token to further authenticate against endpoints you preffer

If you ever find your self in a situation where you need to authenticate against the CSP REST API as app + user, here is a function to do it.

Be aware that the function does require a credential object, but when you atuhenticate against AAD the password is decoded and sent in the post request.

Azure monitoring, connecting the dots

Azure Monitoring

Customer tenant setup

SCOM and Azure Integration

Azure Management Packs

Summary

Wrapping up

Share this:

Resource health through Azure Rest API

Get Azure AD application token

Get all resource groups and resources

Get the resource availability status

Get Resource health based on resource group

Share this:

Authenticate against Micrsoft Partner Center API using Powershell

Share this: