Powershell Archives - Page 2 of 9

Multi subscription deployment with DevOps and Azure Lighthouse

11/03/202003/09/2020
by Martin Ehrnst

Companies today are rapidly adopting new technology. Adding more pressure on the companies IT-department or the service provider. No matter where the workload runs, governance, security, and deployments are fundamental parts.
Azure Lighthouse came last year, and while it solves a lot of our trouble. Multi subscription deployments aren’t one of them – but it sure makes it more possible!

Azure deployments with Azure DevOps

One of the things that make service providers great at what they do is standardization. For example; making sure all subscriptions (customers in many cases) have the same set of baseline Azure policies.

Azure DevOps has multiple ways to deploy resources in Azure or other places. How do we deploy the same Azure template to multiple subscriptions using the same pipeline? In our case, I solved this with some existing features, forward-thinking and PowerShell magic.

Azure DevOps service connection with Azure Lighthouse

Multi subscription deployments with Azure DevOps is not a built-in feature. With Azure Lighthouse it became a little bit easier but will require some work.

First, you must set up a service connection and allow that to access one of your internal subscriptions. In Azure DevOps service connections are bound to one subscription.

For this service connection to be capable of multi subscription deployments, it will need access to your customer’s subscriptions. This can be solved through delegated resource management and Azure Lighthouse.
In my case, I had a group with contributor access. And I could add the SPN to that group. Otherwise, you will have to update your current delegation.

Repository structure

Everything you need, including a YAML pipeline is available on GitHub, but I will walk you through how and why I set it up.

I needed to create not only a solution to deploy one resource to multiple subscriptions. I also needed to deploy multiple ARM templates.

For this purpose, I had the option to create one large PowerShell script with complex logic. Or, I could reuse the same script for every deployment. I chose option two. My code repository now looks something like this

ARM-Templates (folder)
- storage-Account (folder)
  - azuredeploy.json
  - azuredeploy.parameters.json
  - deploy.ps1
- another-resource (folder)
  - azuredeploy.json
  - azuredeploy.parameters.json
  - deploy.ps1
- […]
azure-pipelines.yaml

PowerShell magic?

In regular deployments, we can use built-in tasks in the pipeline and deploy directly. For multi subscription deployment, PowerShell is my weapon of choice.

To people with PowerShell competency. The script used is fairly simple;

Connect to Azure
Retrieve the subscriptions
Iterate and deploy the ARM template(s) to each customer subscriptions.

Below is a short example. Showing the core in my deployment script (deploy.ps1)

$deploymentName = "Multi-sub-deployment"
$deploymentLocation = "westeurope"
$templateFile = ".ARM-Templatesstorage-Accountazuredeploy.json"
$templateParameterFile = ".ARM-Templatesstorage-Accountazuredeploy.parameters.json"

# getting all subscriptions
$subscriptions = Get-AzSubscription | Where-Object { $_.Id -NotIn $excludedSubs }

foreach ($subscription in $subscriptions) {
        
    # set context to the current subscription
    $subscriptionId = $subscription.id
    Set-AzContext -SubscriptionId $subscriptionId

    # deploy the arm template
    New-AzSubscriptionDeployment -Name $deploymentName -Location $deploymentLocation `
        -TemplateParameterFile $templateParameterFile -TemplateFile $templateFile
}

Multi subscription deployment, build pipeline

Although my repository contains a YAML pipeline, you don’t have to use it. To be honest, I’m not sure I like them. I used too much time trying to wrap my head around it. And at this point, it seems unfinished from the Azure DevOps side. Therefore, I will show you how to set up your pipeline to support multi subscription deployments using the classic method.

kick off with the classic mode, and chose the empty job on the template page. We could probably discuss if we even need multiple pipelines for this. But it doesn’t hurt, and it will be easier for the next person if we do this by the book.

For the build pipeline, I am renaming my job to something meaningful, and add one single task. Publish pipeline artifact

After you save and run the build. An artifact should be produced. The result should look something like this

Multi subscription deployment, release pipeline

After a successful build (or in this case copy files), it is time to create our release pipeline. When using YAML, the two pipelines are combined. Not at all confusing for someone like me, who not that many months ago, didn’t know anything at all about this stuff.

Once again, I start off with an empty job, before I add my artifact from my build pipeline. I also renamed the first stage to “resource deployment”.

Now it’s the matter of adding tasks to our job, and it’s here you will need the service connection that you added earlier. The task we are working with is the Azure PowerShell task. And for multi subscription deployment, it is the only task you’ll need. Below is my task configuration

For some reason, it takes a few tries before pipelines want to work. It might be because of lat hours, or a law created by some guy named Murphy. Anyway, once it’s up and running you shole be able to see something similar to my output below;

2020-03-10T17:13:12.2503976Z ## Az module initialization Complete
2020-03-10T17:13:12.2517976Z ## Beginning Script Execution
2020-03-10T17:13:12.2532801Z Generating script.
2020-03-10T17:13:12.3293381Z ========================== Starting Command Output ===========================
2020-03-10T17:13:12.3416275Z ##[command]"C:Program FilesPowerShell6pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'd:a_temp112fe6df-8ee0-4a10-b53e-5694a4d34b0f.ps1'"
2020-03-10T17:13:25.0914923Z No subscription specified. Deploying to all subscriptions
2020-03-10T17:13:25.2151812Z 
2020-03-10T17:13:25.2211374Z Name                                     Account             SubscriptionName    Environment         TenantId
2020-03-10T17:13:25.2292766Z ----                                     -------             ----------------    -----------         --------
2020-03-10T17:13:25.2300155Z MVP-Sponsorship (6dca9329-fb22-46cb-826…  MVP-Sponsorship     AzureCloud          22046864-98a9-4a9…
2020-03-10T17:14:01.7100820Z 
2020-03-10T17:14:01.7151741Z Id                      : /subscriptions/6dca9329-fb22-46cb-826c-/providers/Microsoft.Resources/deployments
2020-03-10T17:14:01.7152947Z                           /Multi-sub-deployment
2020-03-10T17:14:01.7154721Z Location                : westeurope
2020-03-10T17:14:01.7159918Z ManagementGroupId       : 
2020-03-10T17:14:01.7161790Z ResourceGroupName       : 
2020-03-10T17:14:01.7162557Z OnErrorDeployment       : 
2020-03-10T17:14:01.7163149Z DeploymentName          : Multi-sub-deployment
2020-03-10T17:14:01.7163805Z CorrelationId           : d49c10f8-0260-49d5-aa8d-08a41591d1a7
2020-03-10T17:14:01.7164463Z ProvisioningState       : Succeeded
2020-03-10T17:14:01.7165107Z Timestamp               : 3/10/2020 5:14:00 PM
2020-03-10T17:14:01.7166059Z Mode                    : Incremental
2020-03-10T17:14:01.7166610Z TemplateLink            : 
2020-03-10T17:14:01.7167216Z TemplateLinkString      : 
2020-03-10T17:14:01.7167793Z DeploymentDebugLogLevel : 
2020-03-10T17:14:01.7168836Z Parameters              : {[rgName, Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable], 
2020-03-10T17:14:01.7169800Z                           [location, Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable], 
2020-03-10T17:14:01.7173723Z                           [storagePrefix, 
2020-03-10T17:14:01.7174379Z                           Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable]}
2020-03-10T17:14:01.7175014Z ParametersString        : 
2020-03-10T17:14:01.7177870Z                           Name             Type                       Value     
2020-03-10T17:14:01.7178513Z                           ===============  =========================  ==========
2020-03-10T17:14:01.7179228Z                           rgName           String                     adatum    
2020-03-10T17:14:01.7179831Z                           location         String                     westeurope
2020-03-10T17:14:01.7180452Z                           storagePrefix    String                     str       
2020-03-10T17:14:01.7181006Z                           
2020-03-10T17:14:01.7181454Z Outputs                 : {}
2020-03-10T17:14:01.7181916Z OutputsString           : 
2020-03-10T17:14:01.7182291Z 
2020-03-10T17:14:02.0220641Z 
2020-03-10T17:14:02.0937716Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop

If you look at the output, you can see that the script set’s context to a subscription, and that there is no subscription specified in the pipeline.

Multi subscription deployment summary

Multi subscription deployments with Azure DevOps is not available as a default. But with a little bit of PowerShell trickery, you got a great solution. For service providers and large enterprises, Azure Lighthouse is now a preferred way to manage resources.

By granting a service principal access to your customer subscriptions (or internal for that matter), and use this SPN as a service connection in your Azure Pipeline. You can use PowerShell to iterate through each subscription and deploy the resources needed.

The beauty with this is that it will work regardless of where your DevOps environment is hosted. You can have separate tenants for Lighthouse, Azure DevOps and workplace.

This post described the following, which is required for multi subscription deployment to work.

Created an SPN in the management tenant
Authorized the service principal through Azure Lighthouse
Created a repository and added our scripts and templates to it
Created pipelines and used the SPN as our service connection
Used PowerShell and the built-in task to iterate through each subscription and perform deployments.

Azure

Microsoft Secure application model

20/01/201920/01/2019
by Martin Ehrnst

Secure application model was released by Microsoft late last year (2018). At that time, I noticed it and didn’t quite understad how it impacted my work. I therfore moved on.

A few weeks ago i discovered what this change actually means.
If you are a Microsoft cloud service provider or a control panel vendor, you will have to change to this new model of authentication soon. Depending on how you deliver apps or how you manage your customer tenants, there’s quite some work to do.

Microsoft is forcing all user accounts with access to CSP with MFA. That is great, but if you (and likely you are) using app + user credentials to access partner center you cannot do this programmatically, as the current method uses password grant. I have written about how that method works here.

The secure application model depends on refresh tokens and access tokens. As a service provider your customers will have to consent to an application getting access to their tenant. When the admin user consent, you will get a code response. This code is used to create a refresh token, which later can be used to access Azure or other resources.
I do not have the mandate to learn you how refresh and access tokens work, but i found the articles on Oauth.com pretty good.

The below picture shows a broad overview of the flow and ‘infrastructure’ required. I suggest you download the document as well.

Token flow in the secure application model

Secure application model infrastructure

i have built these examples with PowerShell to authenticate to a customer tenant using the new model. The model used here assumes you function as a managed service provider. Maintaining your customers Azure tenants and subscriptions. That way you can consent on behalf of your customers. If you provide Azure market place applications, the process is a bit different, but infrastructure wise, were using the same tools.

In my implementation of secure application model I have used the following tools.

Azure KeyVault
Multi tenant Azure AD Application, with access to the APIs you require.
A user able to consent (in my case member of Admin Agents in CSP)
Single tenant AAD app to authenticate against KeyVault

Multi tenant Azure AD application

If you find this blog post interesting, I assume you already have a multi tenant AAD app used in your integration or software delivery.
If not, you can check out my blog post on how to create Azure AD apps using PowerShell

In my case I have one application used to monitor customers workloads in Azure. This application have access to Azure Management APIs

Single tenant web app

This is the additional application needed. This application will represent your system. In my case a monitoring tool. I use this application registration to access key vault where I have stored my refresh token. The refresh token is then used to get an access token from a customer tenant.

Azure Key Vault

We need a secure place to store the refresh token and possibly other stuff down the road. I chose to run with Key Vault. There are multiple blog posts and documentation on how to provision and give permissions in key vault, but remember to give your single tenant application read and write access to secrets.

Using PowerShell and Rest APIs

As I have multiple times before, I chose to run with REST rather than PowerShell modules. Feel free to use modules or SDKs, it just doesen’t work that well in my environment.

Getting consent

I’m sure theres much slicker ways to do this, but I only needed one consent to make our integration work. If you have multiple refresh tokenst etc. I would build some kind of callback service that could handle the consent flow.
Alter the following code to your needs, paste it in your web browser and sgn in with apropriate credentials. In return, you will have recieve a code. Copy and use this in next step.

Azure AD refresh token

Now that you have consent it is time to get a refresh token. This is what you later use to get access tokens from your customer tenants. By default and if used. The refresh token is valid for 90 days. You will have store this in Key Vault or a similar service.
Add your information to the script below to get your refresh token.

Write and retrieve from Key Vault API

Since you don’t want to get a new consent every time, you will need to save your refresh token to a secure place. I chose to run with Key Vault, but feel free to chose what ever software you want. Below are two snippets that will allow to write and retrieve secrets from Azure KeyVault.
You will have to get your key vault URL and the single tenant application id and secret.
That way your application accessing customers tenants, in my case a monitoring system, have it’s own credentials, separated from the credentials aquiring the refresh and access tokens.

Retrieving data from customer tenant

It’s time to connect to your customers tenants. Before doing that, lets summarize. By now you should have the following in place

One multi tenant application with Api access and proper consent.
A key vault with the a refresh token
A single tenant (your integration) application with access to key vault
AppID and access keys for bot application registrations.

Below I have included three examples on how to retrieve data from your customers. The first will get all customers from partner center, second will use the same refresh token to access Microsoft Graph, and the third will access Azure management API’s (Azure Resource Manager). In order for this to work, your multi tenant app must have access to these APIs

Azure

Serverless application with PowerShell: Azure Functions

03/08/2018
by Martin Ehrnst

In part two we will integrate Azure Functions using Powershell triggered by Event Grid to complete the circle in our application infrastructure. If you havent read the previous parts in this blog series, i recommend you start with the introducution.

Azure Functions, what?

Azure’s serverless flagship probably don’t need any introduction at this stage, but in short. Azure Functions allows you to run code or scripts on demand triggered via various sources. Timer triggers, event, webhooks, http post, gets etc. The most common way to run your stuff is by consumption where your code run on spare compute in azure at a very low-cost. If you demand more power, you can also opt in for the app service plan.

Read more about functions and how to get started here.

FYI; the same type of service from AWS is called Lambda and Google offers similar with Cloud Functions

Powershell support

Powershell support in Azure Functions is still in preview, but I have used it many times, without issues. It is slower that all the other languages supported, but that doesent matter in this case. But be aware that azure functions Powershell run in v4. If you want to run C# or Java feel free to do so.

Let’s start by adding a new function app. Things to notice is that the name of your function app, is also the DNS name. App insights is on by default, and I have no idea why you would want to chose OS type. This wasn’t here before…

without wasting too much of your time reading a lot of fill text, lets just create a new function. Remember to enable ‘experimental language support’ this will magically allow Powershell support (among others). Chose the HTTP trigger Powershell.

I am naming my first function “incoming-pizza-order” which we will add the code to and connect to Event Grid.

After creation you will have a function that supports GET and POST request with your name as the input. A “hello world” type of example. Let’s see if we can trigger this from Event Grid. As we did in the previous post. Create a new Event Grid subscription.

Failure, right? Let’s take a look.

Validating event grid subscriptions

Azure event grid accepts all incoming events and will pass them on to any subscribers. To add a layer of security you will have to validate the subscription upon creation. So all subscribers, weather it’s your own incoming webhook solution on premises or in a another cloud, will have to sort of handshake before it creates the subscription. As you see, the error message in Azure isn’t very good, and I searched quite a long time before I reached out to Ling Toh via Twitter. Luckily she pointed me in the right direction.

Upon creation event grid sends a validation request to the subscriber, which looks something like this:

[{ "id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66", "topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "subject": "", "data": { "validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6", "validationUrl": "https://rp-eastus2.eventgrid.azure.net:553/eventsubscriptions/estest/validate?id=B2E34264-7D71-453A-B5FB-B62D0FDC85EE&t=2018-04-26T20:30:54.4538837Z&apiVersion=2018-05-01-preview&token=1BNqCxBBSSE9OnNSfZM4%2b5H9zDegKMY6uJ%2fO2DFRkwQ%3d" }, "eventType": "Microsoft.EventGrid.SubscriptionValidationEvent", "eventTime": "2018-01-25T22:12:19.4556811Z", "metadataVersion": "1", "dataVersion": "1" }]

We will have to return a successful HTTP response (HTTP 200 OK) with the validation code under the property, “validationResponse”

{ "validationResponse": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6" }

All this is described in Microsofts documentation, but how do we do this in a Powershell function? Here’s my modified version of the “hello world/name” example. As you see it grabs the post requests and if it is a validation event from Event Grid it will return an http 200 and the validation response. Copy the entire script in to your function and re-create the subscription in event grid.

Hopefully the subscription registered successfully. Time to play!

Grab the code example from part 1 and fire off an event, in the matter of seconds your function will fire. You can confirm that it’s working by looking at the log and output.

How cool isn’t this? I love that we can use our Ops skills and create true serverless, modern application infrastructure.

What’s next for our Pizza order example?

In my next post, I will share a few more code examples with multiple Azure Functions, sending events through event grid, from the introduction you know that these are the steps:

Customer creates/orders a pizza online
Send orderdata to event grid
Azure function subscribe to the “new order” event
Create a new event when pizza is in oven
Post new event when pizza is cooked and ready for delivery
Create a new event when pizza is sent to customer.

PS: I would like to create a simple GUI at some point, what tools should I use? I’m thinking either Powershell Universal Dashboard by Adam Driscoll or PowerApps. If you have other suggestions, let me know.

Multi subscription deployment with DevOps and Azure Lighthouse

Azure deployments with Azure DevOps

Azure DevOps service connection with Azure Lighthouse

Repository structure

PowerShell magic?

Multi subscription deployment, build pipeline

Multi subscription deployment, release pipeline

Multi subscription deployment summary

Share this:

Microsoft Secure application model

Secure application model infrastructure

Multi tenant Azure AD application

Single tenant web app

Azure Key Vault

Using PowerShell and Rest APIs

Getting consent

Azure AD refresh token

Write and retrieve from Key Vault API

Retrieving data from customer tenant

Share this:

Serverless application with PowerShell: Azure Functions

Azure Functions, what?

Powershell support

Validating event grid subscriptions

What’s next for our Pizza order example?

Share this: