CSP Archives - adatum

Working with Azure Monitor Rest API

29/02/202003/09/2020
by Martin Ehrnst

It might be an edge case, or you are creating your own integration. But chances are, after fiddling around with Azure Monitor, you encounter a situation where you would have work with its API.
Personally, I have numerous situations over the last years, that required me to integrate directly with Azure Management APIs

In this blog post, I will help you get started using the APIs, hopefully making it less intimidating. All code examples will use PowerShell.

Create SPN and assign it to your subscription RBAC

I am using a service principal in order to connect programmatically to Azure. SPNs are created in Azure Active Directory as Application Registrations. You can choose whether you want to do this via the portal, or by using PowerShell.

After creating (or using an existing SPN) grant the application appropriate access. For simplicity, I am using contributor to my subscription. That might be best for you as well, but one should always use the least privileged assignment needed.

Consider the above as a prerequisite for you to continue.

At this point, you should have an application registration, a secret, and a role assignment on your subscription. We can now use this to acquire an access token and connect to Azure Monitor’s REST API.

Connect to Azure Monitor API using PowerShell

Azure Monitor APIs are a part of the Azure Management APIs. I will, therefore, use these names interchangeably. Also keep in mind, that all other APIs under Azure Management will follow the same methods I demonstrate for Azure Monitor.

To query data we need to authenticate. In the example below, I am using client credentials to acquire the access token. Microsoft’s official example is using the ADAL method, connecting with your identity. I have never had the use for this, as I am usually writing integrations service-to-service.
If you are creating an interactive portal and want to leverage the user’s authorization, ADAL (or MSAL) are probably better.

Retrieve Azure Monitor alert rules

I have no idea why you are exploring Azure Monitors API. Providing an integration solution is therefore not possible. But my gut feeling is that alerts and metrics is a good place to start.

When working with alerts, we need to work with multiple endpoints. Depending on what you are working on, these are the most common;

Alert rules*
Alert incidents
Metric Alerts
Metric Alert status

We can start with one of the basics, retrieve the current configured alert rules. To do that, we need to know what kind it is. The classic alert rules (old type) use a single endpoint, while the current use it’s own.

Below I have included three endpoints and a screenshot. As you can see, all the information that you expect is to be found in the output. From here we can start to explore the alert rule by accessing its properties.

Azure monitor rest api metric alert rule output

Get resource metrics from Azure Monitors API

Metrics is another fundamental in monitoring. When we work with the API in the context of metrics. You can explore the available metrics for each resource type by using the Metric Definitions endpoint.

Actual metrics values require a bit more when it comes to the actual query. The official documentation describes everything pretty well, but I have provided an example for a VM below. This example shows the basics of how you get data from one metric and one VM. You can add multiple metrics to one query, and do additional filtering using the OData filter.

Manage alerts, updating status, etc.

Viewing configured alert rules, looking at disk metrics for a VM. What about alerts. The actual things that send you emails- can we work with them using this API? Yes, you can.

Like I said. Providing an integration solution in this blog post isn’t possible. but most integrations I have seen with Azure Monitor or other monitoring solutions have had some kind of functionality to handle active alerts. Personally, I have created one for SquaredUp earlier, where we could acknowledge alerts in Azure Monitor as well as our on-premises SCOM installation.

Before we wrap up. Let’s take a look at how we can interact with an active alert. I have configured a very naggy alert rule, creating a lot of noise, and I want to change the status of those alerts. Armed with PowerShell and the alerts management endpoints everything is possible.

Summary

This blog post has covered the basics regarding the Azure Monitor REST API and PowerShell. With the examples above and the official documentation, you can start creating your own solutions and integrations.

While we have only covered how to get data out of Azure Monitor, you should know it’s also possible to inject data. By using the HTTP data collector API and the Metric store possibilities are ‘endless’.

Integrations ideas

Alert remediation/handling from a ticketing system
Dashboarding with third-party or custom web integration
Teams/Slack/IM connector
Custom application metrics or logs

In my examples, I have purposely not included how new alert rules are created, as I believe this should be done through ARM. If that is your use case, you should know it is possible and fully supported.

This blog post was originally published in November 2017. Rewritten for Azure Spring Clean 2020 and to reflect changes to Azure Monitor API

Azure

Microsoft Secure application model

20/01/201920/01/2019
by Martin Ehrnst

Secure application model was released by Microsoft late last year (2018). At that time, I noticed it and didn’t quite understad how it impacted my work. I therfore moved on.

A few weeks ago i discovered what this change actually means.
If you are a Microsoft cloud service provider or a control panel vendor, you will have to change to this new model of authentication soon. Depending on how you deliver apps or how you manage your customer tenants, there’s quite some work to do.

Microsoft is forcing all user accounts with access to CSP with MFA. That is great, but if you (and likely you are) using app + user credentials to access partner center you cannot do this programmatically, as the current method uses password grant. I have written about how that method works here.

The secure application model depends on refresh tokens and access tokens. As a service provider your customers will have to consent to an application getting access to their tenant. When the admin user consent, you will get a code response. This code is used to create a refresh token, which later can be used to access Azure or other resources.
I do not have the mandate to learn you how refresh and access tokens work, but i found the articles on Oauth.com pretty good.

The below picture shows a broad overview of the flow and ‘infrastructure’ required. I suggest you download the document as well.

Token flow in the secure application model

Secure application model infrastructure

i have built these examples with PowerShell to authenticate to a customer tenant using the new model. The model used here assumes you function as a managed service provider. Maintaining your customers Azure tenants and subscriptions. That way you can consent on behalf of your customers. If you provide Azure market place applications, the process is a bit different, but infrastructure wise, were using the same tools.

In my implementation of secure application model I have used the following tools.

Azure KeyVault
Multi tenant Azure AD Application, with access to the APIs you require.
A user able to consent (in my case member of Admin Agents in CSP)
Single tenant AAD app to authenticate against KeyVault

Multi tenant Azure AD application

If you find this blog post interesting, I assume you already have a multi tenant AAD app used in your integration or software delivery.
If not, you can check out my blog post on how to create Azure AD apps using PowerShell

In my case I have one application used to monitor customers workloads in Azure. This application have access to Azure Management APIs

Single tenant web app

This is the additional application needed. This application will represent your system. In my case a monitoring tool. I use this application registration to access key vault where I have stored my refresh token. The refresh token is then used to get an access token from a customer tenant.

Azure Key Vault

We need a secure place to store the refresh token and possibly other stuff down the road. I chose to run with Key Vault. There are multiple blog posts and documentation on how to provision and give permissions in key vault, but remember to give your single tenant application read and write access to secrets.

Using PowerShell and Rest APIs

As I have multiple times before, I chose to run with REST rather than PowerShell modules. Feel free to use modules or SDKs, it just doesen’t work that well in my environment.

Getting consent

I’m sure theres much slicker ways to do this, but I only needed one consent to make our integration work. If you have multiple refresh tokenst etc. I would build some kind of callback service that could handle the consent flow.
Alter the following code to your needs, paste it in your web browser and sgn in with apropriate credentials. In return, you will have recieve a code. Copy and use this in next step.

Azure AD refresh token

Now that you have consent it is time to get a refresh token. This is what you later use to get access tokens from your customer tenants. By default and if used. The refresh token is valid for 90 days. You will have store this in Key Vault or a similar service.
Add your information to the script below to get your refresh token.

Write and retrieve from Key Vault API

Since you don’t want to get a new consent every time, you will need to save your refresh token to a secure place. I chose to run with Key Vault, but feel free to chose what ever software you want. Below are two snippets that will allow to write and retrieve secrets from Azure KeyVault.
You will have to get your key vault URL and the single tenant application id and secret.
That way your application accessing customers tenants, in my case a monitoring system, have it’s own credentials, separated from the credentials aquiring the refresh and access tokens.

Retrieving data from customer tenant

It’s time to connect to your customers tenants. Before doing that, lets summarize. By now you should have the following in place

One multi tenant application with Api access and proper consent.
A key vault with the a refresh token
A single tenant (your integration) application with access to key vault
AppID and access keys for bot application registrations.

Below I have included three examples on how to retrieve data from your customers. The first will get all customers from partner center, second will use the same refresh token to access Microsoft Graph, and the third will access Azure management API’s (Azure Resource Manager). In order for this to work, your multi tenant app must have access to these APIs

Azure

Azure monitoring, connecting the dots

04/01/201822/01/2018
by Martin Ehrnst

Azure Monitoring

Welcome to the continuing saga on how to monitor your customers Azure tenants being a service provider. Previously we have covered how to authenticate against Microsoft CSP, using Azure Resource Health API with Powershell and more.

This post is all about connecting the dots. We are far away from finished, but things are moving in this project and at the time of writing, we have two separate projects going.
The first one is focused on creating a single pane of glass for all our customers’ workflows. This involves custom coding and management pack development for SCOM. The second one, which this post will cover, is how we have designed each customer tenant and how we plan to use built-in Azure monitoring functionality.

Customer tenant setup

Working for a service provider we need to construct Azure tenants by taking in to account that we are going to manage cloud resources, so using many cloud features makes a lot of sense. The challenge ist that we always have to think about how we can integrate with an existing deployment and work with monitoring solutions on premises.

When we first started out this project we looked in-to what have been done before, and most of the examples we found wouldn’t scale to our requirements or used OMS/Log Analytics only. We wanted to use our SCOM environment for alert handling, dashboard and platform health as SCOM is already integrated with customer portals, CMDBs and more. We will discuss more on that later in this blog post.

Things are moving very fast in Azure, we have changed our inital customer tenant setup twice before we found a structure we believe is future friendly.
When a customer sign up for an Azure Subscription, we populate their tenant with a default monitoring resource group and a OMS/Log Analytics workspace (LA). Along with the default LA workspace we add the Azure Activity Log, Web Apps and Office 365 solutions as standard.
For “bread and butter” type of Azure Resources, such as compute and web apps we setup the same type of monitoring regime we provide for on-premise resources, but we use alerts in Azure Monitor. This approach works well for Azure Resources which do not have existing, custom Log Analytics solutions and searches to provide health state. This means that VMs deployed using our custom ARM template will also include Monitor Alerts such as “CPU Usage % above 95” and “Web app response time above x”. In conjunction with Azure Monitor we use Azure Resource Health wich will provide health state data regardless of resource type, and custom alerts in monitor or Log Analytics.

Below is a (not so detailed) illustration on our default tenant.

SCOM and Azure Integration

We use System Center Operations Manager (SCOM) as our main monitoring platform for operating systems and applications. As SCOM is already integrated with our ticketing system, CMDB and other internal tools it seems reasonable to provide insight to application and workloads running in Azure on the same monitoring platform. That way we optimally can provide a single pane of glass in to the on premise, hybrid and cloud only workloads.

Azure Management Packs

To get monitoring data in to our on prem SCOM we looked in to two major options.

Option #1:
The official Azure Management pack from Microsoft. he official MP discovery process/adding new tenants cannot be automated. It relies on a GUI where you sign it to the tenant etc. neither does it provide any “umbrella” functionality for companies enrolled in the CSP program.

Option #2:
Daniele Grandini’s Azure/OMS management pack. Daniele’s management packs provide insight to Log Analytics, Azure Backup and Automation, but relies on the official Microsoft MP for initial discovery. Daniele’s management packs focuses on the solutions within the “Monitoring + management” (formerly known as OMS) space in Azure. Since much of the alerting features from OMS/Log Analytics are moving to Azure Monitor, I reached out to Daniele and asked if he had looked in to creating a management pack for that. He had looked a little in to it, but was also concerned about the rapid changes. Unfortunately this MP is bound to the initial discovery from the official Azure MP. A service provider managing several hundred tenants (and growing) cannot have that limitation. I hope to be able to help Daniele with the upcoming Azure Monitor MP.

Here’s where our problems started. I wanted to discover all our manged tenants automatically. Take advantage of being a CSP we set out to create our own management pack(s). I have create one management pack for the CSP platform that integrate with the Partner Center API (see example in this blog post) to do the initial discovery. Tenants and subscriptions are populated as objects in SCOM. Further, using a Partner Center Managed Application we can pre-consent access to all managed tenants. That means we can use this applications credentials to authenticate against each of our managed tenants, by-passing the limitation within the official management pack. All resources are the created as object with a hosting relationship to resourcegroup, subscription and tenant. Basic monitoring is done through Azure Resource Health API.

Below is a diagram showing the structure of our CSP management pack

Credentials used to authenticate against partner center and the Azure tenants is provided through SCOM RunAs accounts.

Our next step in SCOM and Azure integration is to create an Azure Monitor Management pack that reference the CSP management pack. This will provide the more enriched monitoring provided by Azure Monitor. Due to many recent changes to the monitor platform I have decided to wait and see where we end up. At the time of writing Azure Monitor have two new alert features in preview and none of their API’s are officially documented – i will come back with examples when I have something tangible.

Summary

To provide effective monitoring as a service provider for customers which span on-prem and cloud environments, we recommend the following:

For “bread and butter” monitoring use a combination of SCOM and Azure Monitor
If in the CSP program. Create a management pack using CSP rest API’s (hopefully I can share our MP later) combined with a custom Azure Monitor MP
Not a CSP? Look in to a combination of the official MP and Daniele’s management packs.
Deploy Log Analytics as default to all tenants. This will give you an advantage when customers require custom solutions and log sources.

Wrapping up

All service providers do their monitoring differently, but hopefully you have gotten some ideas on how you can do yours. Our solution is far from being finished, but I feel we have a structure that are future proof (the modern type of future). Hopefully we can share the SCOM management packs later, but feel free to contact me on specifics. Just remember I cannot share the MP itself at this point in time.

Until further notice, this will be the closing post on how you can do Azure Monitoring as a service provider.

Big thanks to Kevin Green and Cameron Fuller for providing feedback and to reach out to other community friends on my behalf.

Azure

Authenticate against Micrsoft Partner Center API using Powershell

07/09/201719/12/2019
by Martin Ehrnst

Update 04.01.2019:
While the method described in this post still work. Microsoft are moving to what they call secure app model. Meaning that password grant is deprecated and you will need to use a refresh token model. I have written a new blog post, explaining the new model.

If you’re not familiar with the Microsoft Cloud Service Provider program it’s in short a program to easier let service providers manage their customers tenants and subscriptions within Azure and Office 365 from a centralized platform.

Apart from a very limited web portal it have a set of API’s and SDK’s to build your own solutions – wich I assume is prefered from Microsoft and the service provider. For a project I needed to authenticate against the REST API using Powershell and then retrieve some information about each tenant, who would have thought that could be so much work

Here’s what I said.

That’s fine, I will have it to you in an hour.

For your reference, this is the API I am working with: Partner Center Swagger

An hour later I did have authentication in place, but I was unable to retrieve any information from our customers. After digging through the documentation I found that the customer endpoints required “App + User Authentication” where I had only authenticated with AppId and App Secret.

After spending too much time decifer the C# examples on how you authenticate with app and user against the CSP Rest API i finally had a working Powershell function.

These are the steps required

Generate a token from Azure AD by calling https://login.microsoft.com/tenant-name/oauth/token
- Specified with the resource you want to access (partner center api), client id, username and password, correct grant type and scope
Use the AAD token to authenticate against partnercenter/generatetoken and recieve a correct User + App jwt_token
Use the jwt token to further authenticate against endpoints you preffer

If you ever find your self in a situation where you need to authenticate against the CSP REST API as app + user, here is a function to do it.

Be aware that the function does require a credential object, but when you atuhenticate against AAD the password is decoded and sent in the post request.

Working with Azure Monitor Rest API

Create SPN and assign it to your subscription RBAC

Connect to Azure Monitor API using PowerShell

Retrieve Azure Monitor alert rules

Get resource metrics from Azure Monitors API

Manage alerts, updating status, etc.

Summary

Integrations ideas

Share this:

Microsoft Secure application model

Secure application model infrastructure

Multi tenant Azure AD application

Single tenant web app

Azure Key Vault

Using PowerShell and Rest APIs

Getting consent

Azure AD refresh token

Write and retrieve from Key Vault API

Retrieving data from customer tenant

Share this:

Azure monitoring, connecting the dots

Azure Monitoring

Customer tenant setup

SCOM and Azure Integration

Azure Management Packs

Summary

Wrapping up

Share this:

Authenticate against Micrsoft Partner Center API using Powershell

Share this: