Azure Archives

Serverless application with PowerShell: Azure Functions

03/08/2018
by Martin Ehrnst

In part two we will integrate Azure Functions using Powershell triggered by Event Grid to complete the circle in our application infrastructure. If you havent read the previous parts in this blog series, i recommend you start with the introducution.

Azure Functions, what?

Azure’s serverless flagship probably don’t need any introduction at this stage, but in short. Azure Functions allows you to run code or scripts on demand triggered via various sources. Timer triggers, event, webhooks, http post, gets etc. The most common way to run your stuff is by consumption where your code run on spare compute in azure at a very low-cost. If you demand more power, you can also opt in for the app service plan.

Read more about functions and how to get started here.

FYI; the same type of service from AWS is called Lambda and Google offers similar with Cloud Functions

Powershell support

Powershell support in Azure Functions is still in preview, but I have used it many times, without issues. It is slower that all the other languages supported, but that doesent matter in this case. But be aware that azure functions Powershell run in v4. If you want to run C# or Java feel free to do so.

Let’s start by adding a new function app. Things to notice is that the name of your function app, is also the DNS name. App insights is on by default, and I have no idea why you would want to chose OS type. This wasn’t here before…

without wasting too much of your time reading a lot of fill text, lets just create a new function. Remember to enable ‘experimental language support’ this will magically allow Powershell support (among others). Chose the HTTP trigger Powershell.

I am naming my first function “incoming-pizza-order” which we will add the code to and connect to Event Grid.

After creation you will have a function that supports GET and POST request with your name as the input. A “hello world” type of example. Let’s see if we can trigger this from Event Grid. As we did in the previous post. Create a new Event Grid subscription.

Failure, right? Let’s take a look.

Validating event grid subscriptions

Azure event grid accepts all incoming events and will pass them on to any subscribers. To add a layer of security you will have to validate the subscription upon creation. So all subscribers, weather it’s your own incoming webhook solution on premises or in a another cloud, will have to sort of handshake before it creates the subscription. As you see, the error message in Azure isn’t very good, and I searched quite a long time before I reached out to Ling Toh via Twitter. Luckily she pointed me in the right direction.

Upon creation event grid sends a validation request to the subscriber, which looks something like this:

[{ "id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66", "topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "subject": "", "data": { "validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6", "validationUrl": "https://rp-eastus2.eventgrid.azure.net:553/eventsubscriptions/estest/validate?id=B2E34264-7D71-453A-B5FB-B62D0FDC85EE&t=2018-04-26T20:30:54.4538837Z&apiVersion=2018-05-01-preview&token=1BNqCxBBSSE9OnNSfZM4%2b5H9zDegKMY6uJ%2fO2DFRkwQ%3d" }, "eventType": "Microsoft.EventGrid.SubscriptionValidationEvent", "eventTime": "2018-01-25T22:12:19.4556811Z", "metadataVersion": "1", "dataVersion": "1" }]

We will have to return a successful HTTP response (HTTP 200 OK) with the validation code under the property, “validationResponse”

{ "validationResponse": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6" }

All this is described in Microsofts documentation, but how do we do this in a Powershell function? Here’s my modified version of the “hello world/name” example. As you see it grabs the post requests and if it is a validation event from Event Grid it will return an http 200 and the validation response. Copy the entire script in to your function and re-create the subscription in event grid.

Hopefully the subscription registered successfully. Time to play!

Grab the code example from part 1 and fire off an event, in the matter of seconds your function will fire. You can confirm that it’s working by looking at the log and output.

How cool isn’t this? I love that we can use our Ops skills and create true serverless, modern application infrastructure.

What’s next for our Pizza order example?

In my next post, I will share a few more code examples with multiple Azure Functions, sending events through event grid, from the introduction you know that these are the steps:

Customer creates/orders a pizza online
Send orderdata to event grid
Azure function subscribe to the “new order” event
Create a new event when pizza is in oven
Post new event when pizza is cooked and ready for delivery
Create a new event when pizza is sent to customer.

PS: I would like to create a simple GUI at some point, what tools should I use? I’m thinking either Powershell Universal Dashboard by Adam Driscoll or PowerApps. If you have other suggestions, let me know.

Automation

Serverless application with PowerShell: Azure Event Grid

30/05/201829/05/2018
by Martin Ehrnst

This is part one of my blog series describing how you can create a serverless application infrastructure using PowerShell. The scope and history behind, covered in the introduction post. If you haven’t read that I recommend doing so.

Azure Event Grid

In my serverless infrastructure design, I have chosen to use Azure Event Grid as a hub for the other components in play. The other components will write and subscribe to events/messages in Event Grid and perform their given task as ‘their’ event arrives. Event grid fit’s the concept perfectly as you act on state changes. You can publish as many events you like, but you don’t need to subscribe to every one. A lot of Azures PaaS offerings are already fully integrated which makes it easy to get started.

Other event based offerings in Azure are Service Bus and Event Hub. You can read a little on the differences here

Set up Event Grid custom topic

I am sure you are able to create resources in Azure so here’s a quick one. Feel free to use what ever you want for deployment, but for the purpose of screenshots, I’m using the portal. Just remember that the name you chose will be the url of your endpoint as well.

After the custom topic is successfully deployed, you are given the option to subscribe to events. Before doing that we are taking a break. Besides a graph showing incoming events you have no good way to see your events as they arrive. This makes it way more difficult to debug why stuff isn’t working. While writing this post I researched for a good way to solve this and stumbled across this blog post by David Barkol. He created a web app that lets you view events as they arrive. You can deploy it directly from GitHub using ARM. Do this while your coffee is brewing. And come back here in 10 minutes 🙂

Subscribe to Event Grid topic

Welcome back… If you have deployed the web app you can follow and create a subscription subscribing to all events. In a production scenario, it’s likely to have scoped your subscribers, but in order for Event grid viewer to function i’ll subscribe to everything.

Once deployed your overview page will look something like this. Showing all subscriptions within your topic and their corresponding metrics. Now lets try to send a custom event using powershell.

Send event to event grid using PowerShell

Basically, event grid accepts post request to an API endpoint, which is the name of your topic plus a suffix and the API version. your full URI looks something like this; https://adatumblogdemo.northeurope-1.eventgrid.azure.net/api/events?api-version=2018-01-01

To authenticate we will pass an ‘aeg-sas-key’ within our request header. You will find the keys for authentication in the keys pane. Further event grid events has a schema you willl need to follow, meaning that all post requests will need to have some properties defined and your custom event data will be in a data property. The basics of it all looks like this

[ { "topic": string, "subject": string, "id": string, "eventType": string, "eventTime": string, "data":{ "custom": "data" }, "dataVersion": string, "metadataVersion": string } ]

More details on the schema can be found on Msft Docs

If we combine everything we know this far, we can use the script below to post messages to event grid. Please add your own url and access key. Everything else should work.

Assuming no error messages, jump in to your event grid viewer to view the raw content like this;

A corresponding graph in the Azure portal could look something like this. Here you will see incoming events and how many matches your subscriptions in the same graph. You can also see individual metrics per subscription.

Summary

In this part of my blog series on how to create a serverless application infrastructure using PowerShell. We have walked through how to set up a custom Event Grid topic and how to post messages or event to it using standard PowerShell commandlets. We also deployed a custom web application capable of reading our incoming events in real time.

In the next part: For part two we are going to create an Azure Function running PowertShell that subscribes and send Event Grid messages. This is key to build up our infrastructure wich consists of multiple Azure Functions in order to work.

Automation

Creating a serverless application with Powershell: Introduction

08/05/201807/08/2018
by Martin Ehrnst

Welcome, fellow IT pro. You have found the blog series describing how you can create a serverless application infrastructure with Powershell.

If you consider your self being a ‘modern it pro’, you frequently solve complex tasks and automate your work using code. You know what Git is and have heard people talk about continuous integration. If you can relate, let’s label you a ‘modern it pro’.

When I develop solutions that challenge my knowledge, I often seek advice from our developers. (You know, those who are employed to ‘GIT commit’?) Usually they can share some light on the issue and point me in the right direction.
By developing our own solutions, we have a working proof-of-concept to handover to our developers when things grow too big or complex.

The original challenge

The challenge or scope for my POC was to streamline creation of Azure tenants for our customers. We required to create a tenant, assign subscriptions and do some configuration within Azure AD. Many of these steps was already solved with various scripts, but it is time to consolidate and automate the automation. We decided that we wanted to learn more around the integration and ‘serverless’ PAAS offerings available in Azure.
For the purpose of this blog series we are going to build a new application using the same techniques, in a smaller scale. To create our serverless application infrastructure, we make use of the following Azure offerings:

Pizza ordering

I bet you have worked a fair amount of overtime in your career, therefore eaten a lot of pizza as well. Let’s try to build a simple pizza order and delivery process using Event Grid as integration layer and multiple Azure Functions to process the order.

We will break the pizza order and delivery process in to the following steps:

Customer creates/orders a pizza online
Send orderdata to event grid
Azure function subscribe to the “new order” event
Create a new event when pizza is in oven
Post new event when pizza is cooked and ready for delivery
Create a new event when pizza is sent to customer.

Parts in this blog series

Automation

Creating Azure AD Application using Powershell

10/04/201809/04/2018
by Martin Ehrnst

Azure AD Application

Azure AD have something called Application registrations. These are often used to integrate with external services and can provide functionality like Single Sign On to your companies Twitter account. There’s a large selection of applications you can chose from in the Azure Portal, but this post will cover how to create your own application registration using Powershell.

In this scenario we are creating an app that can access Azure Activity Logs, used by our on premise Splunk environment. Since I am doing this across 300 tenants the manual approach isn’t feasible.

High level overview

Create the app using Powershell
Assign the required API access to the new app
Create access key
Create new Azure AD Service Principal for our app (SPN)
Assign ‘Reader’ role to subscription

Create the app using Powershell

This is the easiest part. Azure Powershell have a pretty simple Cmdlet that let’s you create a new application, New-AzureADApplication. The required steps is to Import AzureRM modules and AzureAD modules. After that, connect to Azure AD using

Connect-AzureAD -Credential -TenantId "domain.onmicrosoft.com"

Now you can run New-AzureAdApplication to create a new app, this example shows the required fields.

New-AzureADApplication -DisplayName "Adatum App Demo" -IdentifierUris "https://localhost/AdatumAppDemo" -HomePage "https://localhost/Adatum"

and in returnObjectId AppId DisplayName -------- ----- ----------- 2cd0a284-7b9e-4 34ecfd2a-8f78-38c4a8b0 Adatum App Demo

In the Azure portal we can see our new app registration, but it does not have a service principal, and no API access. If you would have gone through the steps creating the app in the portal it self SPN and a “read basic profile” API permission would be added to your app by default.

Assign required API access

This is where I spent the most time. In order to assign permissions to our Azure AD Application we will need to write a bit of code. Applications can be assigned Application Permissions and Delegated permissions. According to the documentation New-AzureAdApplication Cmdlet takes a parameter “RequiredResourceAccess”. This again wan’t something of ‘Microsoft.Open.AzureAD.Model.RequiredResourceAccess’ as a generic list. Using my favourite search engine I found that I also needed to construct this object and add multiple objects (the permissions) to it.

Luckily my friend Jan Vidar Elven (MVP) had an example. A sligthly different approach, but here’s what I did. My app needed access to Azure Service Management API, manually I added this via the portal and explored the JSON manifest to see what GUIDs was added. Later I found a better solution, exploring the Service Principal (API’s) using Get-AzureADServicePrincipal.

Get-AzureADServicePrincipal -All $true | Where-Object {$_.DisplayName -eq "Windows Azure Service Management API"}

which will return something like this.

ObjectId AppId DisplayName -------- ----- ----------- 28a1249a-c5bb-4c7b-b94d-064ca5bb1952 797f4846-ba00-4fd7-ba43-dac1f8f63013 Windows Azure Service Management API

Then I could start to build my permission object

## Azure Management API $AzureMgmtPrincipal = Get-AzureADServicePrincipal -All $true | Where-Object {$_.DisplayName -eq "Windows Azure Service Management API"} $AzureMgmtAccess = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess" $AzureMgmtAccess.ResourceAppId = $AzureMgmtPrincipal.AppId

If you had used this object with the ‘RequiredResourceAccess’ parameter now, the app would have assigned Windows Azure Service Management API, but no permissions. Just to be clear, this is the permissions I am talking about

You can explore the same permissions using
$AzureMgmtPrincipal.AppRoles for application permissions and $AzureMgmtPrincipal.Oauth2Permissions for delegated permissions. Azure Management only have one delegated permission

PS C:\> $AzureMgmtPrincipal.Oauth2Permissions
AdminConsentDescription : Allows the application to access the Azure Management Service API acting as users in the organization. AdminConsentDisplayName : Access Azure Service Management as organization users (preview) Id : 41094075-9dad-400e-a0bd-54e686782033 IsEnabled : True Type : User UserConsentDescription : Allows the application to access Azure Service Management as you. UserConsentDisplayName : Access Azure Service Management as you (preview) Value : user_impersonation

Who doesen’t love GUIDs?

Now add this permission to a new ‘microsoft.open.azuread.model.resourceAccess’ object

$AzureSvcMgmt = New-Object -TypeName "microsoft.open.azuread.model.resourceAccess" -ArgumentList "41094075-9dad-400e-a0bd-54e686782033", "Scope" before you add that permission object to the ‘resource access object’ created earlier.

$AzureMgmtAccess.ResourceAccess = $AzureSvcMgmt

If you where adding multiple permissions, repeat the above. After you completed this step, our first object $AzureMgmtAccess have an app ID assigned (Azure management api) with the corresponding Resource access object(s)
ResourceAppId ResourceAccess ------------- -------------- 797f4846-ba00-4fd7-ba43-dac1f8f63013 {class ResourceAccess {...

Now you can re-run our first command but add the permission parameter (delete the app created earlier or change the identifier)

New-AzureADApplication -DisplayName "Adatum App Demo" -IdentifierUris "https://localhost/AdatumAppDemo" -HomePage "https://localhost/Adatum" RequiredResourceAccess @($AzureMgmtAccess)

Check your portal now 🙂

Application Access Key

We have to create an access key for us to programmatically logging in via the application. Depending on your scenario, you might want to create multiple keys, but in this example I am sticking with one. Just like in the Azure portal you will specify the validity time for your key. The example below will add a key wich expire in one year.

New-AzureADApplicationPasswordCredential -ObjectId $AdatumDemoApp.ObjectId -CustomKeyIdentifier "Access Key" -EndDate (get-date).AddYears(1)

If you want the key to not be activated immediately, set the appropriate date time by using StartDate parameter.

Create new Azure AD service principal

For our application to access within our tenant, we need to assign a new service principal. The security principle will allow us to access the subscription (or other resources for that matter.) You can read more about security principals for users and services here, Application and service principal objects in Azure Active Directory (Azure AD)

To create the service principal in via Powershell, run the following: New-AzureADServicePrincipal -AppId -Tags @("WindowsAzureActiveDirectoryIntegratedApp")
Pay attention to that tag. It is required if you want the app to show under app integrations in the Azure AD portal view. Documented here

RBAC / IAM subscription access

The app I am creating is used to read Azure activity logs. To be allowed to read these logs, the app must be assigned ‘Read’ permission on the subscription level. To assign a new role, use the New-AzureRmRoleAssignment cmdlet, using ObjectID, RoleDefinitionName and Scope as input parameters.

New-AzureRmRoleAssignment -ObjectId 'SPN objectId' -RoleDefinitionName "Reader" -Scope "/subscriptions/$SubScriptionId"

PS: When you put the complete script together, you will find that the SPN is create async in Azure Resource Manager. Meaning that you will get the output, but the principal isn’t created. To get around this in our provisioning I created a while loop. Not the prettiest you will find, but works for now.

while ($addRole.DisplayName -ne $AzureSplunkAdApp.DisplayName ) { Start-Sleep -Seconds 5 Write-Verbose "Waiting for SPN to create" $addRole = New-AzureRmRoleAssignment -ObjectId $AzureSplunkAdAppAppSPN.ObjectId -RoleDefinitionName "Reader" -Scope "/subscriptions/$SubScriptionId" }

Final script

I have put together a complete script which will create this demo application. No error handling and your risk 🙂

Azure

Azure monitoring, connecting the dots

04/01/201822/01/2018
by Martin Ehrnst

Azure Monitoring

Welcome to the continuing saga on how to monitor your customers Azure tenants being a service provider. Previously we have covered how to authenticate against Microsoft CSP, using Azure Resource Health API with Powershell and more.

This post is all about connecting the dots. We are far away from finished, but things are moving in this project and at the time of writing, we have two separate projects going.
The first one is focused on creating a single pane of glass for all our customers’ workflows. This involves custom coding and management pack development for SCOM. The second one, which this post will cover, is how we have designed each customer tenant and how we plan to use built-in Azure monitoring functionality.

Customer tenant setup

Working for a service provider we need to construct Azure tenants by taking in to account that we are going to manage cloud resources, so using many cloud features makes a lot of sense. The challenge ist that we always have to think about how we can integrate with an existing deployment and work with monitoring solutions on premises.

When we first started out this project we looked in-to what have been done before, and most of the examples we found wouldn’t scale to our requirements or used OMS/Log Analytics only. We wanted to use our SCOM environment for alert handling, dashboard and platform health as SCOM is already integrated with customer portals, CMDBs and more. We will discuss more on that later in this blog post.

Things are moving very fast in Azure, we have changed our inital customer tenant setup twice before we found a structure we believe is future friendly.
When a customer sign up for an Azure Subscription, we populate their tenant with a default monitoring resource group and a OMS/Log Analytics workspace (LA). Along with the default LA workspace we add the Azure Activity Log, Web Apps and Office 365 solutions as standard.
For “bread and butter” type of Azure Resources, such as compute and web apps we setup the same type of monitoring regime we provide for on-premise resources, but we use alerts in Azure Monitor. This approach works well for Azure Resources which do not have existing, custom Log Analytics solutions and searches to provide health state. This means that VMs deployed using our custom ARM template will also include Monitor Alerts such as “CPU Usage % above 95” and “Web app response time above x”. In conjunction with Azure Monitor we use Azure Resource Health wich will provide health state data regardless of resource type, and custom alerts in monitor or Log Analytics.

Below is a (not so detailed) illustration on our default tenant.

SCOM and Azure Integration

We use System Center Operations Manager (SCOM) as our main monitoring platform for operating systems and applications. As SCOM is already integrated with our ticketing system, CMDB and other internal tools it seems reasonable to provide insight to application and workloads running in Azure on the same monitoring platform. That way we optimally can provide a single pane of glass in to the on premise, hybrid and cloud only workloads.

Azure Management Packs

To get monitoring data in to our on prem SCOM we looked in to two major options.

Option #1:
The official Azure Management pack from Microsoft. he official MP discovery process/adding new tenants cannot be automated. It relies on a GUI where you sign it to the tenant etc. neither does it provide any “umbrella” functionality for companies enrolled in the CSP program.

Option #2:
Daniele Grandini’s Azure/OMS management pack. Daniele’s management packs provide insight to Log Analytics, Azure Backup and Automation, but relies on the official Microsoft MP for initial discovery. Daniele’s management packs focuses on the solutions within the “Monitoring + management” (formerly known as OMS) space in Azure. Since much of the alerting features from OMS/Log Analytics are moving to Azure Monitor, I reached out to Daniele and asked if he had looked in to creating a management pack for that. He had looked a little in to it, but was also concerned about the rapid changes. Unfortunately this MP is bound to the initial discovery from the official Azure MP. A service provider managing several hundred tenants (and growing) cannot have that limitation. I hope to be able to help Daniele with the upcoming Azure Monitor MP.

Here’s where our problems started. I wanted to discover all our manged tenants automatically. Take advantage of being a CSP we set out to create our own management pack(s). I have create one management pack for the CSP platform that integrate with the Partner Center API (see example in this blog post) to do the initial discovery. Tenants and subscriptions are populated as objects in SCOM. Further, using a Partner Center Managed Application we can pre-consent access to all managed tenants. That means we can use this applications credentials to authenticate against each of our managed tenants, by-passing the limitation within the official management pack. All resources are the created as object with a hosting relationship to resourcegroup, subscription and tenant. Basic monitoring is done through Azure Resource Health API.

Below is a diagram showing the structure of our CSP management pack

Credentials used to authenticate against partner center and the Azure tenants is provided through SCOM RunAs accounts.

Our next step in SCOM and Azure integration is to create an Azure Monitor Management pack that reference the CSP management pack. This will provide the more enriched monitoring provided by Azure Monitor. Due to many recent changes to the monitor platform I have decided to wait and see where we end up. At the time of writing Azure Monitor have two new alert features in preview and none of their API’s are officially documented – i will come back with examples when I have something tangible.

Summary

To provide effective monitoring as a service provider for customers which span on-prem and cloud environments, we recommend the following:

For “bread and butter” monitoring use a combination of SCOM and Azure Monitor
If in the CSP program. Create a management pack using CSP rest API’s (hopefully I can share our MP later) combined with a custom Azure Monitor MP
Not a CSP? Look in to a combination of the official MP and Daniele’s management packs.
Deploy Log Analytics as default to all tenants. This will give you an advantage when customers require custom solutions and log sources.

Wrapping up

All service providers do their monitoring differently, but hopefully you have gotten some ideas on how you can do yours. Our solution is far from being finished, but I feel we have a structure that are future proof (the modern type of future). Hopefully we can share the SCOM management packs later, but feel free to contact me on specifics. Just remember I cannot share the MP itself at this point in time.

Until further notice, this will be the closing post on how you can do Azure Monitoring as a service provider.

Big thanks to Kevin Green and Cameron Fuller for providing feedback and to reach out to other community friends on my behalf.

Serverless application with PowerShell: Azure Functions

Azure Functions, what?

Powershell support

Validating event grid subscriptions

What’s next for our Pizza order example?

Share this:

Serverless application with PowerShell: Azure Event Grid

Azure Event Grid

Set up Event Grid custom topic

Subscribe to Event Grid topic

Send event to event grid using PowerShell

Summary

Share this:

Creating a serverless application with Powershell: Introduction

The original challenge

Pizza ordering

Parts in this blog series

Share this:

Creating Azure AD Application using Powershell

Azure AD Application

High level overview

Create the app using Powershell

Assign required API access

Application Access Key

Create new Azure AD service principal

RBAC / IAM subscription access

Final script

Share this:

Azure monitoring, connecting the dots

Azure Monitoring

Customer tenant setup

SCOM and Azure Integration

Azure Management Packs

Summary

Wrapping up

Share this: