adatum

Azure Monitor Managed Prometheus

29/11/202229/11/2022
by Martin Ehrnst

Azure Monitor for Prometheus is released in preview in the fall of 2022. Now let’s just get our terminology right. The Prometheus offering is a part of Azure Monitor Metrics. One part of monitor is Logs, and the other one is Metrics. For some time now Prometheus scraping has been available, but the previous implementation utilized the log part of Azure Monitor. This new offering uses the metric store, which is a way better solution, and the latter is partially why I have an interest in checking it out.

Keep in mind that during the time of writing this. Azure Monitor Managed Prometheus is in preview

Why chose a managed Prometheus offering

Generally speaking, if there’s a managed offer, I am keen to see if it’s better than what you can do yourself. Currently, there are multiple managed Prometheus offerings out there, Grafana being one. And Microsoft just jumped in recently with managed Grafana, and now Azure Monitor Managed Prometheus.

In our environment, we introduced Prometheus before any large players had a solid offering, and for a long time, it did not have persistent storage. So if the service was restarted, metrics were lost. That’s not a good solution in the long run. After a while, we jumped both feet in and set up persistent storage using Thanos, backed by Azure storage for long-term retention. However, this comes with a cost. As with any other system, you host yourself, you need to maintain it, and sometimes things go south. Like when I had to clean up our storage. Since we are 100 percent in Azure, it makes total sense to play around with the newly managed Prometheus offering.

Adding managed Prometheus to an existing AKS cluster

To enable managed Prometheus you need an Azure Monitor workspace. Monitor workspace is like a log analytics workspace, but for the Azure monitor metrics store. This is confusing to many (myself included), and I hope Microsoft cleans up the terminology once things are released. I am not going to dive into how you can start from scratch, or how you can create any of this using Azure Bicep. Stanislav Zhelyazkov already went through this in an excellent post with a lot of good comments. Instead, let’s take a look at how Prometheus can be enabled on an existing cluster, and how you can tune your setup.

The backdrop here is that we already had a Prometheus installation in our cluster, and we also had it connected to Log Analytics using Azure Monitor for Containers. I did not want to interfere with any of the existing stuff, so I manually added the preview extension. This will deploy monitor-metrics pods in your kube-system namespace, and a default scraping configuration.

You can now port-forward to any of the ama-metrics pods and see the configuration and targets. If you are familiar with Prometheus, you would probably expect to find the basic, built-in dashboard solution. But that’s not available. This installation is running in agent mode, using the remote write feature to ship the time-series to Azure Monitor. This means Grafana is your simplest solution to query any data.

Tuning the default Prometheus configuration

cAdvisor metrics are good to have. However, the default configuration does not scrape any of our custom application metrics. For that, you need to enable it by writing a Prometheus scraping config.

Many existing Prometheus setups utilize either a Service Monitor or a Pod Monitor. However these types of CRDs is not supported.

Microsoft has provided us with documentation on how to create a prometheus scrape config so if you know your way around Prometheus this is familiar stuff. However, there are some quirks. The files you create is merged with the default config, and if you do not follow the documentation point-by-point you are bound to create something that’s not working.

There are two config maps to be created. One to enable or disable “system” scraping and one for creating your custom scrape configuration. The documentation state you should do a kubectl create […] –from-file prometheus-config and that the name of the file is very important. The name is indeed very important if you follow Microsoft docs. But if you suddenly find your way to the GitHub repository with the example files, a keen eye will see these are Kubernetes manifest-files.

If you (like i did) try to kubectl create any of these files with that very special name you’ll end up with a broken YAML file as it will try to create the manifest for you.

After some support from a Microsoft representative, I understood what they where trying to explain, so I expect the documentation to change pretty soon.

Anyway, instead of following the documentation. I modified the K8S manifest files and did a kubectl apply which will create the manifest just how you wrote it (and how the github examples look). Below you can see a custom scraping configuration. This example will scrape all pods with annotation prometheus.io/scrape: true.

kind: ConfigMap
apiVersion: v1
data:  
  prometheus-config: |-
    global:
      scrape_interval: 30s
      scrape_timeout: 10s
    scrape_configs:
    - job_name: testjob
      honor_labels: true
      scrape_interval: 30s
      scheme: http
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - target_label: pod_name
        source_labels: [__meta_kubernetes_pod_name]

  debug-mode: |-
    enabled = true
metadata:
  name: ama-metrics-prometheus-config
  namespace: kube-system

We got metrics!

In my Azure hosted Grafana I am now able to query and display our custom metrics. As well as the default cAdvisor data. However, after some time I noticed we had some drops in the collected metrics. I tried to find the same drops in our other Prometheus/Thanos setup but there everything looked normal. Checking the logs gave me the answer.

"ContentLengthLimitExceeded\",\"Message\":\"Maximum allowed content length: 1048576 bytes (1 MB). Provided content length: 25849524

We are producing too large time series, and data is not shipped to Azure Monitor. Due to this we also have pretty high memory usage for the ama-metrics pod, which I am able to check using our other Prometheus instance.

Summary

With this limitation, we cannot use Azure Monitor managed Prometheus for our production environment. The PoC we did was set up in our pre-production environment. I has the same applications, producing the same metrics as our production AKS cluster.

I have spoken with Microsoft about our findings, and I hope the content-length issue will be fixed. As the Azure setup is somehow using Prometheus remote write feature or an abbreviation of it, I assume this will be fixed backend for everyone, or be a configurable tuning option.

On the upside, due to the current limitation in Azure managed Prometheus we managed to tune our existing scraping configuration. By removing unused time-series and labels, we reduced the memory consumption by 1GB. Unfortunately, this tuning was not enough to get under 1MB. Hopefully, Microsoft will fix this, as I am really keen on dropping a self-host Thanos setup in favor Azure Monitor Prometheus

Azure Active Directory

GitHub actions federated identity with Azure AD

24/08/202224/08/2022
by Martin Ehrnst

A long-awaited feature for many is the ability to authenticate with Azure in a GitHub workflow. For a long time, this has been an integrated feature in Azure DevOps, and for some time GitHub actions have had the ability to authenticate using Azure Application registrations and secrets. But now, we can use federated credentials which utilize OIDC- This will completely remove the cumbersome task of rotating app secrets.

Configure an Azure Application with federated identity

At my company, I have decided we should look into moving pipelines from Azure DevOps to GitHub. Federated identity is now Generally Available (GA) which means we are allowed to work with a production solution in mind. So how do you set it up? my suggestion is to follow the official guide. Below is a script that will configure an Azure AD application with GitHub authentication enabled. The script assumes you use a branching strategy. However, you can change this to be any of the other entity types, altering the federationSubcject variable.

	# creates an appregistration in Azure AD and connects it with a github repo
	# use as an example only

	[CmdletBinding()]
	param (
	[Parameter(Mandatory)]
	[string]
	$gitHubRepoName,
	[Parameter(Mandatory)]
	[string]
	$teamName,
	[Parameter(Mandatory)]
	[string]
	$branchName = "main"
	)

	$organization = "myorgltd"
	$appRegistrationName = "gh-$teamName-action-sp"
	$appRegistrationDescription = "Used by $teamName for enabeling GitHub actions with Azure AD OIDC authentication"
	$audience = "api://AzureADTokenExchange"
	$ghIssuer = "https://token.actions.githubusercontent.com/"
	$federationName = "action-$gitHubRepoName"
	$federationSubject = "repo:$($organization)/$($gitHubRepoName):ref:refs/heads/$($branchName)"

	# check if app exists
	$GhAdApp = $null
	$GhAdApp = Get-AzADApplication –DisplayName $appRegistrationName

	if ($GhAdApp) {
	Write-Output "App registration already exist. Adding new credential for specified repo $gitHubRepoName"
	$federatedCred = New-AzADAppFederatedCredential –ApplicationObjectId $GhAdApp.Id `
	–Audience $audience –Issuer $ghIssuer –Subject $federationSubject `
	–Name $federationName –Description "Used to authenticate pipeline in $gitHubRepoName"
	}

	# if app does not eist
	else {
	Write-Output "adding new app registration for $teamName with credentials for $gitHubRepoName"

	$GhAdApp = New-AzADApplication –DisplayName $appRegistrationName –Description $appRegistrationDescription
	$federatedCred = New-AzADAppFederatedCredential –ApplicationObjectId $GhAdApp.Id `
	–Audience $audience –Issuer $ghIssuer –Subject $federationSubject `
	–Name $federationName –Description "Used to authenticate pipeline in $gitHubRepoName"
	}

	$credentialObject = [PSCustomObject]@{
	ApplicationName = "$($GhAdApp.DisplayName)"
	ApplicationObjectId = "$($GhAdApp.Id)"
	}

	Write-Output $credentialObject \| ConvertTo-Json

view raw create-adappgithub.ps1 hosted with ❤ by GitHub

Configure repository secrets in GitHub using PowerShell

As you probably understand, I need to automate this entire process. If our developers are moving from DevOps to Github workflows, it has to be as easy as it can be. No one will move unless they see some benefits, and onboarding is just as easy as the current DevOps setup. I am pretty familiar with Azure as an identity platform, especially with app registrations and enterprise apps. So the above script was done quickly. On the other hand, I am not that familiar with GitHub APIs. I know there’s a CLI, but for various reasons, I need to look into the API, more specifically the repository secrets API. What you can see from the first part of the documentation is that you need to encrypt the secret before you make the put request.

Creates or updates an organization secret with an encrypted value. Encrypt your secret using LibSodium.
GitHub api docs

Now I was in trouble. Should I write everything in .NET, or is it possible to solve this using PowerShell? For sure I would be better off using PowerShell, at least in the PoC phase. First I tried to load the .NET sodium DLLs in my script, but it did not work, and the approach is not that delightful. Luckily with some search engine work, I found a module that wraps Sodium, created specifically for creating secrets in GitHub. Thanks, Tyler! See my example script below.

	# add secrets to repo

	import-module PSSodium

	$ghToken = "ghp_"
	$headers = @{Authorization = "token " + $ghToken}

	$secret = "fb4ca569-c690-4391-96d9-928e7a8fd7ff"
	$repoName = "myrepo"
	$organization = "myorgltd"

	Invoke-RestMethod –Method get –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets" –Headers $headers


	$publicKey = (Invoke-RestMethod –Method get –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets/public-key" –Headers $headers)

	$encryptedSecret = ConvertTo-SodiumEncryptedString –Text $secret –PublicKey $($publicKey.key)

	$secretBody = @"
	{
	"encrypted_value": "$encryptedSecret",
	"key_id": "$($publicKey.key_id)"
	}
	"@
	Invoke-RestMethod –Method Put –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets/AZURE_TENANT_ID" –Headers $headers –body $secretBody

view raw create-ghsecret.ps1 hosted with ❤ by GitHub

What about the pipeline?

Next, you need a working pipeline. Lucky for you the documentation is way better when I first looked at this (before GA), so the example works just fine as it is. Anyway, there are a couple of things to point out. Namely the permissions and the setting for login without a subscription ID.

	name: 'Federated identity test'
	on:
	push:
	branches:
	– main

	permissions:
	id-token: write
	contents: read

	jobs:
	build-and-publish:
	runs-on: ubuntu-latest
	steps:
	– name: checkout
	uses: actions/checkout@main

	– name: azure login
	uses: azure/login@v1
	with:
	client-id: ${{ secrets.AZURE_CLIENT_ID }}
	tenant-id: ${{ secrets.AZURE_TENANT_ID }}
	allow-no-subscriptions: true

view raw workflow.yaml hosted with ❤ by GitHub

Summary

I hope this short post on how to configure Azure and GitHub with federated identity helps you, and that it provided some more information than the official documentation does. If anything please reach out. I now have my scripts in place, and can start the full automation and migration from Azure DevOps pipelines.

Azure Bicep

Speaking at Nordic Infrastructure Conference

18/05/202218/05/2022
by Martin Ehrnst

Nordic Infrastructure Conference is back! This is NICs tenth anniversary, and I am glad to say I am once again able to speak at this conference.

This year I have one session on Azure Bicep, where I will go through (almost) everything you need to know in order to be productive + some bonus tricks and real world scenarios from working with Bicep.
As always with NIC, there’s less slides and more demos!

Azure

Track changes to Azure resources

23/02/202223/02/2022
by Martin Ehrnst

Changes to your Azure resources are quite common, and they are difficult to identify. Until now. Microsoft recently released (in preview) the ability to detect change events using Azure Resource Graph. Meaning you do not have to decipher the administrative events in the activity log of your resources. Much like the change tracking solution for Azure VMs.

How does resource changes work in resource graph?

Resource changes are stored in the resource graph under a new table called ResourceChanges. This table is populated from the activity log and removes the complexity we earlier had with matching correlationId, changeId, and so on.

Note:

Resource changes only reflect data in the Resources table in resource graph

What can I do with this?

There are a lot of reasons and possibility that comes with this feature. Microsoft themself mention

Incident handeling – what happened prior to the incident
CMDB update on changes
Azure policy initiated changes

The first reason is probably a good place to start. In my next post, I will show you how to add an azure monitor alert rule, action group, and an Azure function that can pull relevant change information from the impacted resource.

Azure

How to move Azure blobs up the path

25/01/202225/01/2022
by Martin Ehrnst

This is the short, but for me, pretty intense story from when I uploaded 900 blobs of one Gb each to the wrong path in a storage container. Eventually I was able to move these files using azcopy and PowerShell

Thanos persistent Prometheus metrics

In our Azure Kubernetes environment (AKS) we use Prometheus and Thanos for application metrics. Thanos allow us to use Azure Storage for long term retention and high availability Prometheus setup. The other day I was challenged with deleting a series of metrics causing high cardinality. Meaning that a lot of new series of data was written due to a parameter being inserted during scraping.

The way Thanos works is that it takes raw prometheus data, downsamples it and upload it to Azure Storage for long term retention. Each time this process runs, it will create a new blob. In our production environment we had around 900 blobs and 900gb of data.

Thanos has a built in tool to rewrite these blobs and remove the metric we wanted, which seemed easy enough to do, but we had no idea when the problem first started, so I had to analyze, rewrite and upload all the data. It all seemed to work fine, util I discovered no metrics where available. It turned out that the tool I used inherited my local path and uploaded all the modified data to <guid>/chunks/c:/users/appdata/local/[...]/00001.blob

So no matter how satisfied I was, all the data was useless as thanos expected the files to be under <guid>/chunks/00001. On the bright side, all data was there, so the challenge was to move the files from <guid>/chunks/c:/users/appdata/local/[...] to <guid>/chunks/. From the two pictures below you can see the folder structure. Going trough a download and upload approach was the last thing I wanted to do.

AzCopy and PowerShell to the rescue

I already knew my way around azcopy. But I did not know the process actually run on the Azure backbone if you copy within or between storage accounts. Luckily my dear Twitter friends was there to help where I failed to read the documentation.

To perform the copy operation I used a combination of Azure Powershell and AzCopy.

Connect
Get all current blobs
Filter them
Actually copy
Second loop to delete

Below is my complete script. This could be way smarter but I quickly put it together to get the job done.

	## connect to storage using SAS

	$storageName = ""
	$sasToken = ""
	$container = ""

	$ctx = New-AzureStorageContext –StorageAccountName $storageName –SasToken $sasToken
	# get all the blobs
	$blobs = Get-AzureStorageBlob –Container $container –Context $ctx

	# a date to filter on
	$date = (get-date –Date 20.01.2022 –AsUTC)

	# filter the blobs for date and where name has /c:/..
	$blobsToModify = $blobs \| where { ($_.LastModified.DateTime -ge $date) -and ($_.LastModified.DateTime -le $date.AddHours(24)) -and ($_.Name -like "/chunks/C:/Users/") }

	# loop through the blobs
	# get the original folder name and the blob name with some splitting
	foreach ($blob in $blobsToModify) {

	$blobtoMove = $blob.name
	$original = $blob.Name.split("/",2)[0] # trim to original name
	$newBlob = $blob.Name.split("/")[-1] # trim to original chunk name

	# actually copy
	./azcopy.exe copy "https://$storageName.blob.core.windows.net/thanos/$blobToMove$sasToken" "https://$storageName.blob.core.windows.net/thanos/$original/chunks/$newBlob$sasToken" —overwrite=prompt —s2s–preserve–access–tier=false —include–directory–stub=false —recursive —log–level=INFO;
	}

	# antother loop to delete the whole c:/ folder after the chunks of data is moved
	# i have a separate loop as there might be multiple chunks in the folder.
	foreach ($blob in $blobsToModify) {
	$original = $blob.Name.split("/",2)[0] # trim to original name

	./azcopy.exe remove "https://$storageName.blob.core.windows.net/thanos/$original/chunks/C%3A/$sasToken" —from–to=BlobTrash —recursive —log–level=INFO;
	}

view raw azcopy-move.ps1 hosted with ❤ by GitHub

Summary

I hope this helps someone else who accidentaly upload a lot of data to the wrong place. If you by any chance are using Thanos. I filed this as a bug.

Azure Monitor Managed Prometheus

Why chose a managed Prometheus offering

Adding managed Prometheus to an existing AKS cluster

Tuning the default Prometheus configuration

We got metrics!

Summary

Share this:

GitHub actions federated identity with Azure AD

Configure an Azure Application with federated identity

Configure repository secrets in GitHub using PowerShell

What about the pipeline?

Summary

Share this:

Speaking at Nordic Infrastructure Conference

Share this:

Track changes to Azure resources

How does resource changes work in resource graph?

What can I do with this?

Share this:

How to move Azure blobs up the path

Thanos persistent Prometheus metrics

AzCopy and PowerShell to the rescue

Summary

Share this: