adatum

RunAsRadio – App registrations and enterprise apps

25/01/202323/01/2023
by Martin Ehrnst

Back in December, I did a recording with Richard Campbell the host of RunAsRadio. An IT pro podcast that first aired in 2007- way before I knew podcast was a word. For about half an hour, Richard and I talked about Azure application registrations, enterprise apps, and managed identities. What are they, how can you benefit from them, and what are their similarities and differences? Not to mention how easy it is to get lost. We could have talked for another 60 minutes, but we both thought this was enough content for one episode. You can find the show on your favorite podcast listening service.

What is RunAsRadio

RunAs Radio is a weekly Internet Audio Talk Show for IT Professionals working with Microsoft products. RunAsRadio has put up weekly shows since 2007 and is one of the most successful IT pro podcasts still running.
Richard Campbell is your resident creator and host for Runas Radio. Richard started playing with microcomputers in 1977 at the age of 10. He’s really never done anything else since then.

In that time he’s been involved in every level of the PC industry, from manufacturing to sales to development, and into large-scale infrastructure implementation. Richard is best known as the co-host of .NET Rocks!, but is also known as a Microsoft MVP and RD, a well-seasoned consultant, co-owner of the DevIntersection group of conferences, and founder of the charity Humanitarian Toolbox.

Azure

Azure Monitor Managed Prometheus

29/11/202208/01/2023
by Martin Ehrnst

Azure Monitor for Prometheus is released in preview in the fall of 2022. Now let’s just get our terminology right. The Prometheus offering is a part of Azure Monitor Metrics. One part of monitor is Logs, and the other one is Metrics. For some time now Prometheus scraping has been available, but the previous implementation utilized the log part of Azure Monitor. This new offering uses the metric store, which is a way better solution, and the latter is partially why I have an interest in checking it out.

Keep in mind that during the time of writing this. Azure Monitor Managed Prometheus is in preview

Why chose a managed Prometheus offering

Generally speaking, if there’s a managed offer, I am keen to see if it’s better than what you can do yourself. Currently, there are multiple managed Prometheus offerings out there, Grafana being one. And Microsoft just jumped in recently with managed Grafana, and now Azure Monitor Managed Prometheus.

In our environment, we introduced Prometheus before any large players had a solid offering, and for a long time, it did not have persistent storage. So if the service was restarted, metrics were lost. That’s not a good solution in the long run. After a while, we jumped both feet in and set up persistent storage using Thanos, backed by Azure storage for long-term retention. However, this comes with a cost. As with any other system, you host yourself, you need to maintain it, and sometimes things go south. Like when I had to clean up our storage. Since we are 100 percent in Azure, it makes total sense to play around with the newly managed Prometheus offering.

Adding managed Prometheus to an existing AKS cluster

To enable managed Prometheus you need an Azure Monitor workspace. Monitor workspace is like a log analytics workspace, but for the Azure monitor metrics store. This is confusing to many (myself included), and I hope Microsoft cleans up the terminology once things are released. I am not going to dive into how you can start from scratch, or how you can create any of this using Azure Bicep. Stanislav Zhelyazkov already went through this in an excellent post with a lot of good comments. Instead, let’s take a look at how Prometheus can be enabled on an existing cluster, and how you can tune your setup.

The backdrop here is that we already had a Prometheus installation in our cluster, and we also had it connected to Log Analytics using Azure Monitor for Containers. I did not want to interfere with any of the existing stuff, so I manually added the preview extension. This will deploy monitor-metrics pods in your kube-system namespace, and a default scraping configuration.

You can now port-forward to any of the ama-metrics pods and see the configuration and targets. If you are familiar with Prometheus, you would probably expect to find the basic, built-in dashboard solution. But that’s not available. This installation is running in agent mode, using the remote write (or Microsoft’s own abbreviation of it) feature to ship the time series to Azure Monitor. This means Grafana is your simplest solution to query any data.

Tuning the default Prometheus configuration

cAdvisor metrics are good to have. However, the default configuration does not scrape any of our custom application metrics. For that, you need to enable it by writing a Prometheus scraping config.

Many existing Prometheus setups utilize either a Service Monitor or a Pod Monitor. However these types of CRDs is not supported.

Microsoft has provided us with documentation on how to create a prometheus scrape config so if you know your way around Prometheus this is familiar stuff. However, there are some quirks. The files you create is merged with the default config, and if you do not follow the documentation point-by-point you are bound to create something that’s not working.

There are two config maps to be created. One to enable or disable “system” scraping and one for creating your custom scrape configuration. The documentation state you should do a kubectl create […] –from-file prometheus-config and that the name of the file is very important. The name is indeed very important if you follow Microsoft docs. But if you suddenly find your way to the GitHub repository with the example files, a keen eye will see these are Kubernetes manifest-files.

If you (like i did) try to kubectl create any of these files with that very special name you’ll end up with a broken YAML file as it will try to create the manifest for you.

After some support from a Microsoft representative, I understood what they where trying to explain, so I expect the documentation to change pretty soon.

Anyway, instead of following the documentation. I modified the K8S manifest files and did a kubectl apply which will create the manifest just how you wrote it (and how the github examples look). Below you can see a custom scraping configuration. This example will scrape all pods with annotation prometheus.io/scrape: true.

kind: ConfigMap
apiVersion: v1
data:  
  prometheus-config: |-
    global:
      scrape_interval: 30s
      scrape_timeout: 10s
    scrape_configs:
    - job_name: testjob
      honor_labels: true
      scrape_interval: 30s
      scheme: http
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - target_label: pod_name
        source_labels: [__meta_kubernetes_pod_name]

  debug-mode: |-
    enabled = true
metadata:
  name: ama-metrics-prometheus-config
  namespace: kube-system

We got metrics!

During Christmas 2022 Microsoft released a fix for the content length issue mentioned below. I Have confirmed the fix in our clusters and everything works as expected

In my Azure hosted Grafana I am now able to query and display our custom metrics. As well as the default cAdvisor data. However, after some time I noticed we had some drops in the collected metrics. I tried to find the same drops in our other Prometheus/Thanos setup but there everything looked normal. Checking the logs gave me the answer.

"ContentLengthLimitExceeded\",\"Message\":\"Maximum allowed content length: 1048576 bytes (1 MB). Provided content length: 25849524

We are producing too large time series, and data is not shipped to Azure Monitor. Due to this we also have pretty high memory usage for the ama-metrics pod, which I am able to check using our other Prometheus instance.

Azure Monitor Prometheus pricing

During the preview, the service is free of charge. However, the pricing is publicly available.

Feature	Price
Metrics ingestion (preview)	$0.16 / 10 million samples ingested
Metrics queries (preview)¹⁰	$0.10 / 1 billion samples processed

Azure monitor Prometheus’ pricing

USD $0.16 per 10 million metric samples is good compared to the other options out there. But you need to calculate in the queries as well. I suspect in larger environments, dashboards running auto-refresh, etc. this will be a significant part of the total bill. Not to mention that you also need some kind of graphical interface to query your metrics. You could host your own Grafana, or check out Azure Managed Grafana. And remember to calculate the total cost.
Grafana cloud is another option. They offer Prometheus and log ingestion on a per-user basis in their pro plan.

I have yet to sit down and compare pricing between options and will make sure to update or create a new post when we have conducted that work. At first glance, Azure seems favorable, but we will find out once the numbers are crunched.

Summary.

Due to the previous limitation in Azure managed Prometheus we managed to tune our existing scraping configuration. By removing unused time series and labels, we reduced the memory consumption by 1GB. Unfortunately, this tuning was not enough to get under 1MB.
Since a fix was released in December 2022 I will continue my testing with our pre-production cluster. I am really keen to replace the current Thanos setup, and Azure monitor Prometheus looks like a promising option.

Azure Active Directory

GitHub actions federated identity with Azure AD

24/08/202224/08/2022
by Martin Ehrnst

A long-awaited feature for many is the ability to authenticate with Azure in a GitHub workflow. For a long time, this has been an integrated feature in Azure DevOps, and for some time GitHub actions have had the ability to authenticate using Azure Application registrations and secrets. But now, we can use federated credentials which utilize OIDC- This will completely remove the cumbersome task of rotating app secrets.

Configure an Azure Application with federated identity

At my company, I have decided we should look into moving pipelines from Azure DevOps to GitHub. Federated identity is now Generally Available (GA) which means we are allowed to work with a production solution in mind. So how do you set it up? my suggestion is to follow the official guide. Below is a script that will configure an Azure AD application with GitHub authentication enabled. The script assumes you use a branching strategy. However, you can change this to be any of the other entity types, altering the federationSubcject variable.

	# creates an appregistration in Azure AD and connects it with a github repo
	# use as an example only

	[CmdletBinding()]
	param (
	[Parameter(Mandatory)]
	[string]
	$gitHubRepoName,
	[Parameter(Mandatory)]
	[string]
	$teamName,
	[Parameter(Mandatory)]
	[string]
	$branchName = "main"
	)

	$organization = "myorgltd"
	$appRegistrationName = "gh-$teamName-action-sp"
	$appRegistrationDescription = "Used by $teamName for enabeling GitHub actions with Azure AD OIDC authentication"
	$audience = "api://AzureADTokenExchange"
	$ghIssuer = "https://token.actions.githubusercontent.com/"
	$federationName = "action-$gitHubRepoName"
	$federationSubject = "repo:$($organization)/$($gitHubRepoName):ref:refs/heads/$($branchName)"

	# check if app exists
	$GhAdApp = $null
	$GhAdApp = Get-AzADApplication –DisplayName $appRegistrationName

	if ($GhAdApp) {
	Write-Output "App registration already exist. Adding new credential for specified repo $gitHubRepoName"
	$federatedCred = New-AzADAppFederatedCredential –ApplicationObjectId $GhAdApp.Id `
	–Audience $audience –Issuer $ghIssuer –Subject $federationSubject `
	–Name $federationName –Description "Used to authenticate pipeline in $gitHubRepoName"
	}

	# if app does not eist
	else {
	Write-Output "adding new app registration for $teamName with credentials for $gitHubRepoName"

	$GhAdApp = New-AzADApplication –DisplayName $appRegistrationName –Description $appRegistrationDescription
	$federatedCred = New-AzADAppFederatedCredential –ApplicationObjectId $GhAdApp.Id `
	–Audience $audience –Issuer $ghIssuer –Subject $federationSubject `
	–Name $federationName –Description "Used to authenticate pipeline in $gitHubRepoName"
	}

	$credentialObject = [PSCustomObject]@{
	ApplicationName = "$($GhAdApp.DisplayName)"
	ApplicationObjectId = "$($GhAdApp.Id)"
	}

	Write-Output $credentialObject \| ConvertTo-Json

view raw create-adappgithub.ps1 hosted with ❤ by GitHub

Configure repository secrets in GitHub using PowerShell

As you probably understand, I need to automate this entire process. If our developers are moving from DevOps to Github workflows, it has to be as easy as it can be. No one will move unless they see some benefits, and onboarding is just as easy as the current DevOps setup. I am pretty familiar with Azure as an identity platform, especially with app registrations and enterprise apps. So the above script was done quickly. On the other hand, I am not that familiar with GitHub APIs. I know there’s a CLI, but for various reasons, I need to look into the API, more specifically the repository secrets API. What you can see from the first part of the documentation is that you need to encrypt the secret before you make the put request.

Creates or updates an organization secret with an encrypted value. Encrypt your secret using LibSodium.
GitHub api docs

Now I was in trouble. Should I write everything in .NET, or is it possible to solve this using PowerShell? For sure I would be better off using PowerShell, at least in the PoC phase. First I tried to load the .NET sodium DLLs in my script, but it did not work, and the approach is not that delightful. Luckily with some search engine work, I found a module that wraps Sodium, created specifically for creating secrets in GitHub. Thanks, Tyler! See my example script below.

	# add secrets to repo

	import-module PSSodium

	$ghToken = "ghp_"
	$headers = @{Authorization = "token " + $ghToken}

	$secret = "fb4ca569-c690-4391-96d9-928e7a8fd7ff"
	$repoName = "myrepo"
	$organization = "myorgltd"

	Invoke-RestMethod –Method get –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets" –Headers $headers


	$publicKey = (Invoke-RestMethod –Method get –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets/public-key" –Headers $headers)

	$encryptedSecret = ConvertTo-SodiumEncryptedString –Text $secret –PublicKey $($publicKey.key)

	$secretBody = @"
	{
	"encrypted_value": "$encryptedSecret",
	"key_id": "$($publicKey.key_id)"
	}
	"@
	Invoke-RestMethod –Method Put –Uri "https://api.github.com/repos/$($organization)/$($repoName)/actions/secrets/AZURE_TENANT_ID" –Headers $headers –body $secretBody

view raw create-ghsecret.ps1 hosted with ❤ by GitHub

What about the pipeline?

Next, you need a working pipeline. Lucky for you the documentation is way better when I first looked at this (before GA), so the example works just fine as it is. Anyway, there are a couple of things to point out. Namely the permissions and the setting for login without a subscription ID.

	name: 'Federated identity test'
	on:
	push:
	branches:
	– main

	permissions:
	id-token: write
	contents: read

	jobs:
	build-and-publish:
	runs-on: ubuntu-latest
	steps:
	– name: checkout
	uses: actions/checkout@main

	– name: azure login
	uses: azure/login@v1
	with:
	client-id: ${{ secrets.AZURE_CLIENT_ID }}
	tenant-id: ${{ secrets.AZURE_TENANT_ID }}
	allow-no-subscriptions: true

view raw workflow.yaml hosted with ❤ by GitHub

Summary

I hope this short post on how to configure Azure and GitHub with federated identity helps you, and that it provided some more information than the official documentation does. If anything please reach out. I now have my scripts in place, and can start the full automation and migration from Azure DevOps pipelines.

RunAsRadio – App registrations and enterprise apps

What is RunAsRadio

Share this:

Azure Monitor Managed Prometheus

Why chose a managed Prometheus offering

Adding managed Prometheus to an existing AKS cluster

Tuning the default Prometheus configuration

We got metrics!

Azure Monitor Prometheus pricing

Summary.

Share this:

GitHub actions federated identity with Azure AD

Configure an Azure Application with federated identity

Configure repository secrets in GitHub using PowerShell

What about the pipeline?

Summary

Share this: