Azure Bicep
Nordic Infrastructure Conference is back! This is NICs tenth anniversary, and I am glad to say I am once again able to speak at this conference.
This year I have one session on Azure Bicep, where I will go through (almost) everything you need to know in order to be productive + some bonus tricks and real world scenarios from working with Bicep.
As always with NIC, there’s less slides and more demos!
Azure
This is the short, but for me, pretty intense story from when I uploaded 900 blobs of one Gb each to the wrong path in a storage container. Eventually I was able to move these files using azcopy and PowerShell
In our Azure Kubernetes environment (AKS) we use Prometheus and Thanos for application metrics. Thanos allow us to use Azure Storage for long term retention and high availability Prometheus setup. The other day I was challenged with deleting a series of metrics causing high cardinality. Meaning that a lot of new series of data was written due to a parameter being inserted during scraping.
The way Thanos works is that it takes raw prometheus data, downsamples it and upload it to Azure Storage for long term retention. Each time this process runs, it will create a new blob. In our production environment we had around 900 blobs and 900gb of data.
Thanos has a built in tool to rewrite these blobs and remove the metric we wanted, which seemed easy enough to do, but we had no idea when the problem first started, so I had to analyze, rewrite and upload all the data. It all seemed to work fine, util I discovered no metrics where available. It turned out that the tool I used inherited my local path and uploaded all the modified data to <guid>/chunks/c:/users/appdata/local/[...]/00001.blob
So no matter how satisfied I was, all the data was useless as thanos expected the files to be under <guid>/chunks/00001. On the bright side, all data was there, so the challenge was to move the files from <guid>/chunks/c:/users/appdata/local/[...] to <guid>/chunks/. From the two pictures below you can see the folder structure. Going trough a download and upload approach was the last thing I wanted to do.
I already knew my way around azcopy. But I did not know the process actually run on the Azure backbone if you copy within or between storage accounts. Luckily my dear Twitter friends was there to help where I failed to read the documentation.
To perform the copy operation I used a combination of Azure Powershell and AzCopy.
Below is my complete script. This could be way smarter but I quickly put it together to get the job done.
I hope this helps someone else who accidentaly upload a lot of data to the wrong place. If you by any chance are using Thanos. I filed this as a bug.
Azure Active Directory
There’s no secret you can get an Azure AD token and access API resources like Microsoft Graph, Azure Resource Manager (ARM), etc. It’s also pretty straightforward to authenticate a custom API using client credentials. In fact, I have written about how to do that previously where we accessed a custom API built on Azure Functions. Authentication-wise, I also wrote a post on how to access the Azure Monitor REST APIs using client credentials (app registration).
The above examples are fine. But they both use a separate app registration for authenticating against our custom API, the Azure Function, and against ARM to access Azure Monitor. But what if I want to use my own, personal credentials instead of client credentials. For ARM resources, like Azure Monitor, Resource Graph, etc. You can do that already using Azure CLI, or the PowerShell example below.
Connect-AzAccount
Get-AzAccessToken -ResourceUrl "https://management.azure.com"
Wheater you use Az CLI or PowerShell, the output is similar to the above. you can decipher the token using jwt.io. And get a human-readable output.
Always be careful when using services like JWT.io Your token is after all your credentials and can give access to resources.
Instead of specifying ARM as we did above, you can also generate a token against your custom app registration using delegated permissions from Azure CLI or PowerShell. The secret lies in the “expose and API”, or more specifically, “Authorized client applications”.
To allow delegated access and the ability to receive a token from your custom app registration do the following
04b07795-8ddb-461a-bbee-02f9e1bf7b461950a258-227b-4e31-a9cf-717495945fc2
Pull out your favorite shell and change you’re ResourceUrl from management.azure.com to your app id or URI. In my case, this is api://adatum-auth-test-app
After getting the token you can again use JWT.io and see the details. Pay attention to the appId and aud. AppId in this case is Azure PowerShell.
This post has been laying around in my draft for more than a year. But yesterday I got a question from a colleague about this and figured it was time to release it to the masses.
The reason I had it laying in drafts is that I am unsure of the supportability from Microsoft and the potential security vulnerability it may add to your services. However, keep that in mind and use the feature when needed.
If you want to learn more about application registrations, enterprise apps, and managed identities in general. Please read my other post about the topic.