Skip to content
adatum
  • Home
  •  About adatum
  •  Learn Azure Bicep
  •  SCOM Web API
Azure

How to move Azure blobs up the path

  • 25/01/202225/01/2022
  • by Martin Ehrnst

This is the short, but for me, pretty intense story from when I uploaded 900 blobs of one Gb each to the wrong path in a storage container. Eventually I was able to move these files using azcopy and PowerShell

Thanos persistent Prometheus metrics

In our Azure Kubernetes environment (AKS) we use Prometheus and Thanos for application metrics. Thanos allow us to use Azure Storage for long term retention and high availability Prometheus setup. The other day I was challenged with deleting a series of metrics causing high cardinality. Meaning that a lot of new series of data was written due to a parameter being inserted during scraping.

The way Thanos works is that it takes raw prometheus data, downsamples it and upload it to Azure Storage for long term retention. Each time this process runs, it will create a new blob. In our production environment we had around 900 blobs and 900gb of data.

Thanos has a built in tool to rewrite these blobs and remove the metric we wanted, which seemed easy enough to do, but we had no idea when the problem first started, so I had to analyze, rewrite and upload all the data. It all seemed to work fine, util I discovered no metrics where available. It turned out that the tool I used inherited my local path and uploaded all the modified data to <guid>/chunks/c:/users/appdata/local/[...]/00001.blob

So no matter how satisfied I was, all the data was useless as thanos expected the files to be under <guid>/chunks/00001. On the bright side, all data was there, so the challenge was to move the files from <guid>/chunks/c:/users/appdata/local/[...] to <guid>/chunks/. From the two pictures below you can see the folder structure. Going trough a download and upload approach was the last thing I wanted to do.

Azure storage explorer

AzCopy and PowerShell to the rescue

I already knew my way around azcopy. But I did not know the process actually run on the Azure backbone if you copy within or between storage accounts. Luckily my dear Twitter friends was there to help where I failed to read the documentation.

To perform the copy operation I used a combination of Azure Powershell and AzCopy.

  • Connect
  • Get all current blobs
  • Filter them
  • Actually copy
  • Second loop to delete

Below is my complete script. This could be way smarter but I quickly put it together to get the job done.

## connect to storage using SAS
$storageName = ""
$sasToken = ""
$container = ""
$ctx = New-AzureStorageContext –StorageAccountName $storageName –SasToken $sasToken
# get all the blobs
$blobs = Get-AzureStorageBlob –Container $container –Context $ctx
# a date to filter on
$date = (get-date –Date 20.01.2022 –AsUTC)
# filter the blobs for date and where name has /c:/..
$blobsToModify = $blobs | where { ($_.LastModified.DateTime -ge $date) -and ($_.LastModified.DateTime -le $date.AddHours(24)) -and ($_.Name -like "*/chunks/C:/Users/*") }
# loop through the blobs
# get the original folder name and the blob name with some splitting
foreach ($blob in $blobsToModify) {
$blobtoMove = $blob.name
$original = $blob.Name.split("/",2)[0] # trim to original name
$newBlob = $blob.Name.split("/")[-1] # trim to original chunk name
# actually copy
./azcopy.exe copy "https://$storageName.blob.core.windows.net/thanos/$blobToMove$sasToken" "https://$storageName.blob.core.windows.net/thanos/$original/chunks/$newBlob$sasToken" —overwrite=prompt —s2s–preserve–access–tier=false —include–directory–stub=false —recursive —log–level=INFO;
}
# antother loop to delete the whole c:/ folder after the chunks of data is moved
# i have a separate loop as there might be multiple chunks in the folder.
foreach ($blob in $blobsToModify) {
$original = $blob.Name.split("/",2)[0] # trim to original name
./azcopy.exe remove "https://$storageName.blob.core.windows.net/thanos/$original/chunks/C%3A/$sasToken" —from–to=BlobTrash —recursive —log–level=INFO;
}
view raw azcopy-move.ps1 hosted with ❤ by GitHub

Summary

I hope this helps someone else who accidentaly upload a lot of data to the wrong place. If you by any chance are using Thanos. I filed this as a bug.

Share this:

  • LinkedIn
  • Twitter
  • Reddit
Automation

Remediate Azure Policy with PowerShell

  • 11/06/202011/06/2020
  • by Martin Ehrnst

Azure Policy is there to help us with properly governed and secure infrastructure. However, Azure Policy requires management as well.

Lately, I have built a new set of policies to ensure diagnostic logs are forwarded to Azure Monitor Logs. Multiple policies and a policy initiative were deployed to multiple subscriptions and multiple customers. All this was made possible since we manage through Azure Lighthouse.

Automatic remediation of Azure Policy

The challenge faced after deploying the policy was how to remediate them. Since policies with effect ‘deployIfNotExists’ only apply to new or modified resources, I faced the job with clicking in the portal or figure out a way to do this with PowerShell.
I actually started with the portal, as I thought it would be a quick job. After doing one or two subscriptions I realized how much time I would use.

Azure policy compliance state

Given the fact that the imitative it self contained around 50 individual policies, and at the time 19 subscriptions. I figured spending some time in PowerShell was well worth it. There is also a pretty good chance I will find my self in the same situation pretty soon.

Create remediation task with PowerShell

To create a remediation task for a policy set you can use this script. It will connect to your subscription and get all non-compliant policies. Then start a policy remediation task for the individual policies.

Summary

Policies with effect “deployIfNotExist” only work for resources that are updated or created after the policy was applied. To remediate existing resources you will have to create the remediation tasks manually through the portal, or by using PowerShell (and REST API)

By the way, fellow Azure MVP Tao Yang has created everything you need in order to enable these policies your self. Please see GitHub for complete ARM templates. And please help him maintain everything by contributing.

Share this:

  • LinkedIn
  • Twitter
  • Reddit
Azure

Multiple Azure credentials in PowerShell

  • 06/05/202002/09/2020
  • by Martin Ehrnst

Environments in Azure are often separated into multiple subscriptions, in some cases multiple tenants. This sectioning can also result in multiple user accounts, and managing multiple Azure credentials can be challenging. Luckily, Azure PowerShell has this capability called context.

Although the documentation is solid on the topic. It doesn’t necessarily provide the backdrop for when you will need to use this feature.

A few weeks ago I had to move a set of APIs from one instance of Azure API management to another. With PowerShell contexts, I could download the API from the origin and import it to the new instance.

Connecting to multiple Azure environments using context

To hold credential information, like user and subscription. PowerShell uses context objects. By using AzContext comandlets You can have multiple Powershell Azure contexts available in the same PowerShell session. This allows for easy switching between multiple environments and profiles. Including different tenants.

Below is an example of how you can connect to multiple tenants and switch between the credentials and contexts. I recommend using friendly names which will make them easier to identify.

Azure PowerShell context

Share this:

  • LinkedIn
  • Twitter
  • Reddit

Posts navigation

1 2 3 … 9

Popular blog posts

  • Azure Application registrations, Enterprise Apps, and managed identities
  • SCOM 1801 REST API Interfaces
  • Creating Azure AD Application using Powershell
  • Automate Azure DevOps like a boss
  • Access to Blob storage using Managed Identity in Logic Apps - by Nadeem Ahamed

Categories

Automation Azure Azure Active Directory Azure Bicep Azure DevOps Azure Functions Azure Lighthouse Azure Logic Apps Azure Monitor Azure Policy Community Conferences CSP Monitoring DevOps Guest blogs Infrastructure As Code Microsoft CSP MPAuthoring OMS Operations Manager Podcast Powershell Uncategorised Windows Admin Center Windows Server

Follow Martin Ehrnst

  • Twitter
  • LinkedIn

RSS feed RSS - Posts

RSS feed RSS - Comments

Microsoft Azure MVP

Martin Ehrnst Microsoft Azure MVP
Adatum.no use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it. Cookie Policy
Theme by Colorlib Powered by WordPress