Infrastructure as code Archives

Azure Infrastructure as code – Pulumi

10/12/202010/12/2020
by Martin Ehrnst

Infrastructure As Code is here to stay. And all companies work with this in a variety of ways. Recently I changed job, and with that comes new challenges. The team I joined I highly skilled and is responsible for a very complex, and large infrastructure in Azure. A great part of this infrastructure is deployed and maintained using a tool called Pulumi.

My new role does not require me to become a developer creating Apps. But I have been advocating and teaching fellow IT pro’s the importance of embracing developer tools and processes for our infrastructure management tasks. My knowledge around infrastructure as code is with PowerShell, Terraform, and ARM. My C# skills are very limited, although I have some experience.
Pulumi is definitely putting developers first, and I need to step up my game.

What is Pulumi

Azure Resource Manager and Azure Bicep are both domain-specific languages, meaning they only work with Azure. Terraform, is another popular tool (almost a standard), which also has it’s own language (HCL). HCL differs from ARM as it works with more than Azure.

Create, deploy, and manage infrastructure on any cloud using familiar programming languages and tools.
Pulumi

Pulumi on the other hand, use general-purpose programming languages. This means you can deploy and maintain your infrastructure with ‘real programming languages’, like C#, Java, TypeScript, and Go.

How does Pulumi work

Pulumi is a declarative infrastructure as code tool. And it’s core engine will ‘build’ your desired infrastructure, and keep track of its state.

Projects and stacks

You start with something called a Project. The project folder is controlled via a Pulumi.yml file looking something like this, where name and runtime are mandatory.

name: core-infra
runtime: dotnet
description: my very first pulumi project

After creating the project you will need to create a stack. The stack is an instance of your project. For example, staging and production of project core-infra would be two separate stacks.

State management

You might be familiar with this concept already, but if not here’s what’s what;
Pulumi keeps a snapshot of your infrastructure, referred to as ‘state’. This allows Pulumi to delete, create, and change your infrastructure components. But it also means you have to think about where you perform edits (only within the Pulumi stack/project), and where to store your state files.

By default Pulumi will store and manage state with their online service, Pulumi Console.

Getting started with Pulumi for Azure

My short goal for self learning Pulumi is to replicate what I demoed in me and Marcel Zehner’s Live streams on Azure resource manager and infrastructure as code.
for Pulumi I am using this repository

For some reason, I assume you run Windows and CSharp, but if you fancy any of the other options, they are documented as well.

To run Pulumi on Azure you will need to install Pulumi, log in/sign up, install .NET 3.1, and Azure CLI (if you don’t have it already). The process is documented on the getting started page.
I tried to run with .NET 5.0, without any luck, but that might be solved soon.

Your next task is to create your project. In all essence, you run a few commands against an empty folder. This will generate the Pulumi program files and your project metadata files. Below is my configuration

cd C:users\MartinEhrnst\repos\Pulumi\
mkdir 1.ResourceGroup-storageAccount
cd 1.ResourceGroup-storageAccount
pulumi new azure-csharp

After filling in your mandatory project parameters, a getting started code will be generated for you. This will create an Azure resource group and a storage account.
In the above picture, I have changed this slightly to include a storage container, and change some of the default parameters. You can find my latest Pulumi code in this GitHub repo

For those experienced with C#, you can see that Pulumi has classes for the Azure resources. But since this is C#, we can use common coding techniques, like iterations (for-each) to deploy our infrastructure.

Pulumi deployments

If I now want to deploy my infrastructure. I will need to run Pulumi, which translates this code into something Azure Resource Manager can understand. To my knowledge, Pulumi uses the Azure Resource Manager REST APIs to run the deployment.

To deploy the resources, you can follow this guide. In my environment above, this is the code and output from my review.

PS C:\Users\MartinEhrnst\repos\Pulumi\1.ResourceGroup-storageAccount> pulumi up
Previewing update (dev)

View Live:

     Type                         Name                Plan
 +   pulumi:pulumi:Stack          rg-and-storage-dev  create
 +   ├─ azure:core:ResourceGroup  resourceGroup       create
 +   ├─ azure:storage:Account     storage             create
 +   └─ azure:storage:Container   container           create
 
Resources:
    + 4 to create

Do you want to perform this update? details
+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:dev::rg-and-storage::pulumi:pulumi:Stack::rg-and-storage-dev]
    + azure:core/resourceGroup:ResourceGroup: (create)
        [urn=urn:pulumi:dev::rg-and-storage::azure:core/resourceGroup:ResourceGroup::resourceGroup]
        [provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-]
        location  : "norwayeast"
        name      : "rg-PulumiStorage"
    + azure:storage/account:Account: (create)
        [urn=urn:pulumi:dev::rg-and-storage::azure:storage/account:Account::storage]
        [provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        accountKind           : "StorageV2"
        accountReplicationType: "LRS"
        accountTier           : "Standard"
        allowBlobPublicAccess : false
        enableHttpsTrafficOnly: true
        isHnsEnabled          : false
        location              : output<string>
        minTlsVersion         : "TLS1_0"
        name                  : "storage2966fa9"
        resourceGroupName     : "rg-PulumiStorage"
    + azure:storage/container:Container: (create)
        [urn=urn:pulumi:dev::rg-and-storage::azure:storage/container:Container::container]
        [provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        containerAccessType: "private"
        name               : "images"
        storageAccountName : "storageab46f04"

In Azure, I can now see that the storage account and resource group are created. But I cannot find this as deployments. I suspect this has to do with how Pulumi interacts with Azure resource manager. This might not be an issue for you, but if you rely on the deployment plane, you should have given this a thought.

Should you use Pulumi for Azure?

Given my very limited knowledge of the product that is hard for me to answer. But there are things you should consider.
As I said, I have advocated for a few years about the ‘Modern IT pro’. Meaning we need to adopt and use more developer-oriented software and processes, like Git for example.

By using Pulumi you are not only adopting processes, but you also assume your team knows CSharp or any of the other supported languages. If your team consists of IT Pro’s who are beginning to explore the Dev side of the DevOps circle. Pulumi will give you some rough weeks ahead.

On the other hand, if your team is developer heavy, looking into the operations side. Pulumi might be your best choice. As a developer, it must seem alluring to be able to provision infrastructure together with your application code.
However, the responsibility for correct configuration, governance, and security is still the most important for your infrastructure. Can this be done with the same team and codebase, you can definitely consider using Pulumi.

Pulumi ARM template converter

A tool to convert ARM templates to Pulumi already exists. During my initial testing, I had success converting less complex templates, but when I tried to convert a nested template with a Copy loop the tool failed.

I suggest you try it out with your own templates, and since it’s open-sourced, you could always try to improve it your self. If not, the community will at some point.

Azure

Azure Infrastructure As Code video series

28/10/202028/10/2020
by Martin Ehrnst

For weeks Marcel Zehner and I have held four live streams. Covering ‘everything’ related to Infrastructure as code on Azure.

Recording available

In the series, we covered the following topics, and everything is now available on YouTube

Advanced ARM templates
Deployment scripts
Linked and nested ARM templates
ARM template deployment with Pipelines

Azure

Recording available: ARM template deployment scripts

23/10/202023/10/2020
by Martin Ehrnst

In our Infrastructure as code live stream series, Marcel Zehner and I covered a new feature in the Azure IAC space. Namely deployment scripts. With deployment scripts for Azure, you can run PowerShell or Bash (CLI) scripts within your ARM templates.

You may have heard of Custom script extension for Azure Windows VMs, this is sort of the same thing. Making you able to call external systems during deployment, or perform the last-mile configuration to your infrastructure.

Deployment scripts recording

You can find all our recordings on youtube and all code examples on github

Automation

Using Azure pipelines to deploy ARM templates

27/02/201903/09/2020
by Martin Ehrnst

There are many reasons that you would use an Azure Resource Manager (ARM) template to deploy your Azure resources. And there’s equally many ways to deploy this kind of template.

For simple one time deployments of Azure resources I tend to use PowerShell or AZ CLI, but lately I have ben busy trying to create a web api based on Azure Functions using C# and .NET core. For those who know me, that’s not straight foreward.
To make things even more diffucult, I wanted to use Azure DevOps and Pipelines to build and push my code. In order to do that, I had to have some infrastructure in place, so why wouldn’t I use Azure Pipelines to deploy the ARM template as well?

Create ARM template

In order to deploy Azure Resource Manager templates, you will need to create one. ARM templates are based on Json and follows a schema. This schema or rule set if you like, defines how you must structure your template.

There are two forms of deployment options. At a resource group level (most common) and subscription level deployment. Both uses the same kind of Json language but have two different schemas. You’ll find the current resource group deployment schema here (no need to read :))

If you are unfamiliar with Json and ARM templates I recommend you to read Microsofts official documentation and getting started guide which will show how you can find the template from an existing deployment or create a new.

For reference, this is a simple template to deploy a web app, that I copied from Microsofts GitHub repo with a bunch of templates to get you started.

Adding ARM template along with your code

As I already mentioned deploy ARM template in a pipeline makes sense in many ways, especially if you have developed an application. This way you can control the infrastructure along side the code.

My Azure Function uses Cosmos DB and Key Vault as parts of it’s infrastructure. It is all developed in Visual Studio Code, and the whole thing is cutting edge for an old operations guy like me.

In the picture below you’ll see that I have structured my project in to two subfolders. One is holding the ARM templates, and the other one is for the acutal functions code. I did this so i would be able to chose artifacts from the build process in Azure DevOps pipelines. There might be a better way, but when i had everything in the same folder the templates was packed with the code and not available in deployment pipeline.

vscode function project with arm template folder

Set up the build pipeline

Not long ago I learned that there are two types of pipelines. Build pipelines and release pipelines. Build is used to pack up or actually build the application (same as pressing f5 in Visual Studio). Where release are used for pushing the application on to your infrastructure (and deploy the infra).

For reference. In Azure DevOps, my folder structure looks exactly the same

Next move over to pipelines and create a new release pipeline. From here chose your repository (I use Azure repos).

After chosing your repo, ou can chose to start from a template. My project is a .NET core application and I had no idea what was needed to build that, so a template worked nicely. Here you can chose what’s best for you’re project. The important pieces from an ARM template perspective comes in the next steps

Start out by filling in the obvious. A name and a server to do the build. In my case i run with Azure hosted 2017. In an on premises environment I would use private servers.

Add your templates as build artifacts

What you will need to do next is to take those Json files in your arm template catalog, and make them available as part of the build. You will have to add in a “Copy files” step and fill in a few properties. Pay attention to where I chose the source folder and the destination folder which have a variable reference, $(build.artifactstagingdirectory)/arm

Now. Hit save and queue. This will save your new (or edited pipeline) and start a build of your project.

By now you should also have a build running, and if you’re like me, exploring new stuff, you get used to this

In this particular error. The publish task for .NET core is missing www root folder. By trial and error I found that I had to make a few changes to my build pipeline. This is not a prerequisite for running ARM templates with Azure pipelines, but i figured I should add in everything I had to do for this project, both for my reference and for yours if needed.

I removed “publish web projects” as i don’t have an ordinary web project, but an Azure function based API.

When you remove this tick, the task will refere to your project file (.CSPROJ).

Here i should give a shout out to my trusty colleague Emil Kjelsrud for helping out, once again.

Hit save and queue once again. Voila!

Configure the release pipeline with ARM templates

When your build is successful, we’re close to launch our application on to Azure. We do that by configuring a release pipeline. It is possible to combine the two, but I like them separated.

Under Pipelines > Releases hit new. A new wizard will pop up. Either chose one that fit your purpose, or start from scratch. In my case I chose to start with app service deployment. One of the first things you notice is “some settings need attention.”

Required settings in this context are the connection to your Azure environment and subscription. You will need to configure this in order to continue. There are several methods available for connecting your pipelines to Azure. I have my pipeline deploying resources in a tenant and subscription that my user does not have access to. That configuration requires a SPN/App registration in Azure AD with the correct permissions.

After configuring your connection. Modify your pipeline by adding in your required Azure Resource Group Deployment tasks. Here comes a few important bits. Prior to setting up my pipeline, I have a resource group created already. If you want to have the resource group created during deployment. You can use the Azure CLI or Azure powershell tasks (or a template) to do so.

Continue to fill in the required parameters. Remember that I separated my application code and the templates in to separate folders? This is why. I can now chose from the same artifact what i want to deploy, as the build pipeline have them available in their respective folders. Chose the template and your parameters file.

In this Azure DevOps environment we have an extension available that parses all outputs from your ARM templates. This way I can use variables from the previous step in my app service deployment. If you dont have this extension available, you can achieve the same result using PowerShell

Create the release

You’re pipeline is now complete. Save and create a release, the pipeline should now pick up the latest build and deploy your code to Azure. You can follow the whole process in Azure, under resource group > deployment and in Azure DevOps

After a few minutes (or seconds) depending on your configuration, you should have both your infrastructure and the actual application available.

Final thoughts

While I understand many of the concepts around infrastructure as code, continious integration and so on, I don’t always use the correct terms, so thank you for reading.

I hope you also have a greater understanding around the concepts, and also how you can utilize the possibilities available. My thoughts working with projects like this is always bound to the operational side. I know I’m going to use more ARM templates in Azure Pipelines, but probably not along side the actual code, as that’s not my day-to-day work. Again, I see my self working more and more with these Dev tools, and thats why I like to say that this is how we do modern Ops.

I leared a few more tips and tricks during this small project, so expect a few more posts soon.

Azure Infrastructure as code – Pulumi

What is Pulumi

How does Pulumi work

Projects and stacks

State management

Getting started with Pulumi for Azure

Pulumi deployments

Should you use Pulumi for Azure?

Pulumi ARM template converter

Share this:

Azure Infrastructure As Code video series

Recording available

Share this:

Recording available: ARM template deployment scripts

Deployment scripts recording

Share this:

Using Azure pipelines to deploy ARM templates

Create ARM template

Adding ARM template along with your code

Set up the build pipeline

Add your templates as build artifacts

Configure the release pipeline with ARM templates

Create the release

Final thoughts

Share this: