Powershell Archives

How to move Azure blobs up the path

25/01/202225/01/2022
by Martin Ehrnst

This is the short, but for me, pretty intense story from when I uploaded 900 blobs of one Gb each to the wrong path in a storage container. Eventually I was able to move these files using azcopy and PowerShell

Thanos persistent Prometheus metrics

In our Azure Kubernetes environment (AKS) we use Prometheus and Thanos for application metrics. Thanos allow us to use Azure Storage for long term retention and high availability Prometheus setup. The other day I was challenged with deleting a series of metrics causing high cardinality. Meaning that a lot of new series of data was written due to a parameter being inserted during scraping.

The way Thanos works is that it takes raw prometheus data, downsamples it and upload it to Azure Storage for long term retention. Each time this process runs, it will create a new blob. In our production environment we had around 900 blobs and 900gb of data.

Thanos has a built in tool to rewrite these blobs and remove the metric we wanted, which seemed easy enough to do, but we had no idea when the problem first started, so I had to analyze, rewrite and upload all the data. It all seemed to work fine, util I discovered no metrics where available. It turned out that the tool I used inherited my local path and uploaded all the modified data to <guid>/chunks/c:/users/appdata/local/[...]/00001.blob

So no matter how satisfied I was, all the data was useless as thanos expected the files to be under <guid>/chunks/00001. On the bright side, all data was there, so the challenge was to move the files from <guid>/chunks/c:/users/appdata/local/[...] to <guid>/chunks/. From the two pictures below you can see the folder structure. Going trough a download and upload approach was the last thing I wanted to do.

AzCopy and PowerShell to the rescue

I already knew my way around azcopy. But I did not know the process actually run on the Azure backbone if you copy within or between storage accounts. Luckily my dear Twitter friends was there to help where I failed to read the documentation.

To perform the copy operation I used a combination of Azure Powershell and AzCopy.

Connect
Get all current blobs
Filter them
Actually copy
Second loop to delete

Below is my complete script. This could be way smarter but I quickly put it together to get the job done.

	## connect to storage using SAS

	$storageName = ""
	$sasToken = ""
	$container = ""

	$ctx = New-AzureStorageContext –StorageAccountName $storageName –SasToken $sasToken
	# get all the blobs
	$blobs = Get-AzureStorageBlob –Container $container –Context $ctx

	# a date to filter on
	$date = (get-date –Date 20.01.2022 –AsUTC)

	# filter the blobs for date and where name has /c:/..
	$blobsToModify = $blobs \| where { ($_.LastModified.DateTime -ge $date) -and ($_.LastModified.DateTime -le $date.AddHours(24)) -and ($_.Name -like "/chunks/C:/Users/") }

	# loop through the blobs
	# get the original folder name and the blob name with some splitting
	foreach ($blob in $blobsToModify) {

	$blobtoMove = $blob.name
	$original = $blob.Name.split("/",2)[0] # trim to original name
	$newBlob = $blob.Name.split("/")[-1] # trim to original chunk name

	# actually copy
	./azcopy.exe copy "https://$storageName.blob.core.windows.net/thanos/$blobToMove$sasToken" "https://$storageName.blob.core.windows.net/thanos/$original/chunks/$newBlob$sasToken" —overwrite=prompt —s2s–preserve–access–tier=false —include–directory–stub=false —recursive —log–level=INFO;
	}

	# antother loop to delete the whole c:/ folder after the chunks of data is moved
	# i have a separate loop as there might be multiple chunks in the folder.
	foreach ($blob in $blobsToModify) {
	$original = $blob.Name.split("/",2)[0] # trim to original name

	./azcopy.exe remove "https://$storageName.blob.core.windows.net/thanos/$original/chunks/C%3A/$sasToken" —from–to=BlobTrash —recursive —log–level=INFO;
	}

view raw azcopy-move.ps1 hosted with ❤ by GitHub

Summary

I hope this helps someone else who accidentaly upload a lot of data to the wrong place. If you by any chance are using Thanos. I filed this as a bug.

Azure Active Directory

Azure token from a custom app registration

20/01/202220/01/2022
by Martin Ehrnst

There’s no secret you can get an Azure AD token and access API resources like Microsoft Graph, Azure Resource Manager (ARM), etc. It’s also pretty straightforward to authenticate a custom API using client credentials. In fact, I have written about how to do that previously where we accessed a custom API built on Azure Functions. Authentication-wise, I also wrote a post on how to access the Azure Monitor REST APIs using client credentials (app registration).

Get an Azure token with delegated user credentials from a custom API

The above examples are fine. But they both use a separate app registration for authenticating against our custom API, the Azure Function, and against ARM to access Azure Monitor. But what if I want to use my own, personal credentials instead of client credentials. For ARM resources, like Azure Monitor, Resource Graph, etc. You can do that already using Azure CLI, or the PowerShell example below.

Connect-AzAccount
Get-AzAccessToken -ResourceUrl "https://management.azure.com"

Wheater you use Az CLI or PowerShell, the output is similar to the above. you can decipher the token using jwt.io. And get a human-readable output.

Always be careful when using services like JWT.io Your token is after all your credentials and can give access to resources.

App registration expose an API

Instead of specifying ARM as we did above, you can also generate a token against your custom app registration using delegated permissions from Azure CLI or PowerShell. The secret lies in the “expose and API”, or more specifically, “Authorized client applications”.

To allow delegated access and the ability to receive a token from your custom app registration do the following

Make sure your user is allowed to access the app, you can add that in the enterprise app blade.
Create a scope under “expose and API”
Add client application(s) to the scope
- Azure CLI client application ID: 04b07795-8ddb-461a-bbee-02f9e1bf7b46
- Azure PowerShell client application ID: 1950a258-227b-4e31-a9cf-717495945fc2

Get access token from custom API using Azure CLI or PowerShell

Pull out your favorite shell and change you’re ResourceUrl from management.azure.com to your app id or URI. In my case, this is api://adatum-auth-test-app

After getting the token you can again use JWT.io and see the details. Pay attention to the appId and aud. AppId in this case is Azure PowerShell.

Final words

This post has been laying around in my draft for more than a year. But yesterday I got a question from a colleague about this and figured it was time to release it to the masses.

The reason I had it laying in drafts is that I am unsure of the supportability from Microsoft and the potential security vulnerability it may add to your services. However, keep that in mind and use the feature when needed.
If you want to learn more about application registrations, enterprise apps, and managed identities in general. Please read my other post about the topic.

Azure DevOps

Automate Azure DevOps like a boss

23/06/202123/06/2021
by Martin Ehrnst

Over the past weeks, I have worked on automation within Azure DevOps. One of the tasks was to create a self-service tool for our developers to create new projects where all pre-requisites were taken care of. That means the project itself, service connections (ARM) to access our Azure Kubernetes Service (AKS), Azure AD groups for accessing the project, and so on.

For the Azure AD part, I had mostly everything in place already. Service connections for Azure AD are essentially Azure application registrations, and Azure AD groups are easily created using New-AzADGroup or Microsoft Graph API. However, my experience with Azure DevOps API was and still is limited.

Creating the project

Creating the project itself is pretty straightforward. Below is a PowerShell snippet to do so. I used a service account and a Personal Access Token (PAT) with the following access permissions to do everything in my code.

Graph, read/manage
Identity, read/manage
Member Entitlement Management, read/write
Project and team, read/write/manage
Security, manage
Service connections, read/manage/query

Assigning permissions using Azure DevOps API

You don’t get far with just a project. Someone needs to have access to it as well. Since I try to create a paved road for the DevOps users I needed to add users to the DevOps groups inside the project. Here I spent some significant time, trying to understand how. In my routine, I already create three AAD groups for Project Administrators, contributors, and readers. We want to have one single pane for managing access to company resources, and Azure AD is the place.

In my opinion, access control in Azure DevOps is a nightmare, but lets not focus on that. I want to add each AAD group in to it’s corresponding Azure DevOps group.

To do that you use Azure DevOps Graph and the Groups endpoints. What you can see in the Microsoft example is that you need to perform a post operation against https://vssps.dev.azure.com/{organization}/_apis/graph/groups?groupDescriptors=vssgp.Uy0xLTktMTU1MTM3NDI0NS0yMTc2MDU2ODQ4LTMyODAwNzczODUtMjM4NDQzMDA4Mi0yNTc3Njk3NTA4LTEtNzM0NDQ1NzM2LTQwNzkyNzIyNjgtMzA0NzY5MjIyMy0yMjg2NTY0ODM0&api-version=6.1-preview.1

Azure DevOps Descriptor

From the documentation, you can see the request to add AAD groups into a project group like Project Administrators you need to specify a groupDescriptor. I could not find any good information on how to get this information, and as you can see, the request does not specify the project, only at the organization level.

Now, after some digging, I figured I could fetch this information with some additional calls to the API.

Get your project descriptor, by using the projectId as your descriptor…
Get all groups within the project, using the actual project descriptor
Filter the groups you want from the above request
Pass the Azure AD group objectId in the body and POST against /graph/groups?groupDescriptors=

Modify Azure DevOps project features

In our case, we do not use all features in Azure DevOps. I, therefore, wanted to turn off some features like boards. Making it a bit more difficult for our developers to start using features that we have other tools to support. I couldn’t find any official endpoint documented, so I reached out on Twitter to get inputs. Of course fellow MVP Rob Bos had some time to inspect the portal calls and came up with the solution. Basically, after the project is created, you need to make additional calls to an old API to flip the switch for each feature you do not use.

Adding ARM service connections to project using DevOps API

This part is quite well documented. So after creating the app registrations and service principals in Azure AD. You need to create the service connection(s) in the project. In my case I had more than one service connection, so I had to run this snippet multiple times, as I could not find a batch operation for it.

I think that pre-provisioning of the service connections are a good idea. This feature is quite complex to grasp for many, as the built in wizard cause more confusion than it helps. Especially when the users try to create a new SC, and they do not have access to create App registrations in Azure, or to set permissions on the subscription. Then the entire wizard will fail, but you might end up with ghost SPNs in Azure…

I said it was properly documented, but if you take a close look. The endpoint is called servicendpoint and not serviceconnection which is what it is called in the portal.

Automation solutions for Azure DevOps

In my examples I use the native Azure DevOps APIs and PowerShell. This worked best in our solution. However, you can also use Azure CLI or Azure DevOps provider for Terraform.

Since we do not use Terraform for any IAC, I weren’t keen to start using it just for Azure DevOps. Azure CLI have some limitations since it only allow interactive login, or a semi-solution for the PAT. I suggest you see what solution that works for you. But my examples above do at least work.

Summary

In this post I showed how I solved Azure DevOps project creation. The code above only show half the solution as it is very domain specific. However, I hope it helps you along the way. Creating the actual project is only one part of it. You need to create a paved road for the users. Therefore, make sure the users are able to access their projects and deploy resources. Automating the setup will ensure that every project is set up equally and supporting future requests from your developers will be much easier.

Azure

Multiple Azure credentials in PowerShell

06/05/202002/09/2020
by Martin Ehrnst

Environments in Azure are often separated into multiple subscriptions, in some cases multiple tenants. This sectioning can also result in multiple user accounts, and managing multiple Azure credentials can be challenging. Luckily, Azure PowerShell has this capability called context.

Although the documentation is solid on the topic. It doesn’t necessarily provide the backdrop for when you will need to use this feature.

A few weeks ago I had to move a set of APIs from one instance of Azure API management to another. With PowerShell contexts, I could download the API from the origin and import it to the new instance.

Connecting to multiple Azure environments using context

To hold credential information, like user and subscription. PowerShell uses context objects. By using AzContext comandlets You can have multiple Powershell Azure contexts available in the same PowerShell session. This allows for easy switching between multiple environments and profiles. Including different tenants.

Below is an example of how you can connect to multiple tenants and switch between the credentials and contexts. I recommend using friendly names which will make them easier to identify.

Azure PowerShell context

Community

Speaking at Microsoft Tech Summit Oslo

14/11/201815/11/2018
by Martin Ehrnst

Microsoft Tech Summit is arriving in Oslo December 6 2018.
I am so lucky that I will have a talk on how you can leverage Azure serverless offerings to automate your business processes. All using your existing Ops skills.

The cloud as the established foundation allows the real transformative technologies such as Artificial Intelligence start to transform the way we live and work. The Microsoft Tech summit is your key for inspiration, matching your innovative app ideas with the latest evolution of the Microsoft Cloud platform, also supporting Open Source solutions.

How to move Azure blobs up the path

Thanos persistent Prometheus metrics

AzCopy and PowerShell to the rescue

Summary

Share this:

Azure token from a custom app registration

Get an Azure token with delegated user credentials from a custom API

App registration expose an API

Get access token from custom API using Azure CLI or PowerShell

Final words

Share this:

Automate Azure DevOps like a boss

Creating the project

Assigning permissions using Azure DevOps API

Azure DevOps Descriptor

Modify Azure DevOps project features

Adding ARM service connections to project using DevOps API

Automation solutions for Azure DevOps

Summary

Share this:

Multiple Azure credentials in PowerShell

Connecting to multiple Azure environments using context

Azure PowerShell context

Share this:

Speaking at Microsoft Tech Summit Oslo

Share this: