Azure Bicep
Nordic Infrastructure Conference is back! This is NICs tenth anniversary, and I am glad to say I am once again able to speak at this conference.
This year I have one session on Azure Bicep, where I will go through (almost) everything you need to know in order to be productive + some bonus tricks and real world scenarios from working with Bicep.
As always with NIC, there’s less slides and more demos!
Azure Bicep
A common problem for many organizations is to share and consume infrastructure templates. Many ended up with a storage account in Azure, but that had some limitations around versioning and sharing of secrets. Directly consuming templates from a Git repository is another option. However, that’s not exactly problem-free either. For example, what happens if a colleague makes changes, and you reference the template without knowing what has changed. In the best case, a missing parameter will fail your pipeline. In the worst case, you have downtime.
The key concept of having Bicep modules in a common store is for everyone inside your organisation to use these modules when they’re provisioning infrastructure. Picture this scenario. Multiple development team is often using the same type of resource, like Azure SQL, Azure Functions, storage accounts, etc. Your organization likely have a few governance rules applied. Like tagging strategy, allowed SKU, different configuration for test and production and so on. Pre-created and easially consumable modules taking care of this is what you need. Azure Container Registry for Bicep files is now available.
With Azure bicep version 0.4.1008 you have a built-in option to publish your Bicep modules to a private registry. The private registry is not a new resource type, in fact, you are uploading your Bicep files to Azure Container Registry which allows you to leverage versioning which will make sure you do not break templates for everyone each time people make changes. Once your Bicep module is uploaded to ACR everyone with permission to pull images can use your modules.
A shared bicep module is used in the same way as a local module, but instead of the local path, you specify the URL and version of the file within the registry. More on that later.
Assuming you already have one Bicep module or a set of them, you only need to provision a container registry. To push the “image” you need acrPush permissions and to consume you need acrPull. Below is the syntax used for uploading a bicep file to ACR.
az bicep publish storage.bicep --target br:exampleregistry.azurecr.io/bicep/modules/storage:v1I am using this code base against my existing registry, so my command to upload is as follows;
bicep publish .\Bicep\3.modules\sql.bicep --target "br:acrbicepehrnst.azurecr.io/modules/azuresql/sql:v0.1"No response is given on successful upload, but to make sure everything is alright we can confirm with this PowerShell command, which will list all repositories in your registry.
Get-AzContainerRegistryRepository -RegistryName acrbicepehrnstIncluding modules from a private registry is as easy as using local modules. With Bicep extension enabled in VSCode, you also get validation of the remote modules.
pssst… if you by any change use EMACS you can have the same through LSP and Bicep Lang server
var tags = {
'owner': 'Martin Ehrnst'
'purpose': 'Bicep demo'
}
module SQL 'br:acrbicepehrnst.azurecr.io/modules/azuresql/sql:v0.1' = {
name: 'sqlDeploy'
params: {
databaseName: 'moduletest'
dbAdId: '8776fb6e-5de0-408c-be03-c17a67b079d0'
dbAdLoginName: 'name@company.com'
env: 'prod'
tags: tags
}
} In this post I have showed you the core concept, how you upload and how you consume the modules. To me this is only half the story. In my next post I will go through how we can put everything inside a pipeline and add a better versioning to the modules.
Azure Bicep private registry is probably here to stay. Upuntil now it is the best solution to share infrastructure templates within an organization.
Azure Bicep
With infrastructure as code, we strive to parameterize, re-use as much as possible, and make our code as modular as possible. As your application infrastructure grows, it might become too much work to have everything decoupled and modular. But for ad-hoc deployments, “bread and butter” resources we can make things very agile. Azure Bicep modules, variables and parameters are here to help
At the moment I am collaborating on rewriting some of our infrastructures to use Azure Bicep. This infrastructure is set up multiple times for different environments like test and production, but they are mostly identical when it comes to the resources used. Therefore all resources now exist as Azure Bicep Modules. The modules themselves handle the different configurations based on the environment being provisioned. test environments using a smaller number of nodes in the AKS for example.
Bicep modules are here to help us abstract the complexity of our deployment templates. Make it easier to re-use and share the templates across environments, applications, and teams. It is totally up to you how you create and structure your modules, as all .bicep files can be used as a module. You can also include as many individual resources you like. In the end, bicep will combine all module files and create a nested deployment.
If you have ever worked with nested deployments in ARM templates, you will be glad Bicep modules now exist.
Below I have added a fairly simple module that will provision a storage account with the inputted storageName parameter. In my main.bicep file I will ask for this parameter, and pass it down to my module. The main file is actually more complex than the module file, as its also provisioning a resource group that the storage account will be in.
With the above example in mind. How can we move from that to a more complex and re-usable scenario? First of all, we need to accept a few more inputs in our main file. But the complexity will need to be handled inside our storage account module. Our goal is to re-use the module regardless of the environment being provisioned. Therefore we need to handle scenarios like name, storage SKU, and so on. I use a storage account as an example here, but the concept is the same regardless of resource types.
To extend my module’s modularity (sorry) I am expanding the required input parameters, but I also added an object variable to hold my configuration settings for the different environments. Looking at the storage account module now, you can see the new variable, and also how I get the data from within the object variable by using the inputted name for environment.
I also added more smartness to the module, by using the toLower function and concatenating the storage account name with a unique string based on our resource group. This way we move the complexity of naming the storage account from the user to the code it self. Storage accounts only accepts lower case in its name, and it needs to be globally unique.
The main file changed slightly as well. As you can see, I added environment parameter, with two allowed values, prod and test. I also have a tags variable that I pass down to the module as input parameter together with the environment. The resource group name now has environment represented in the name as well.
Tags and naming conventions is of course very domain specific, but this example show how you can use the different functions available in Azure Bicep to create a more elastic code for your specific environments where resource types is shared.
In the above examples, I showed how you can use Bicep Modules, parameters and variables to re-use your templates cross multiple environments with different properties. If you like me, have multiple environments that on the technology side is identical, but properties like network addresses, virtual machine families, etc differs. You can use modules and custom variable to handle it.
Stay tuned for more Bicep posts in the near future.
Infrastructure As Code
Infrastructure As Code is here to stay. And all companies work with this in a variety of ways. Recently I changed job, and with that comes new challenges. The team I joined I highly skilled and is responsible for a very complex, and large infrastructure in Azure. A great part of this infrastructure is deployed and maintained using a tool called Pulumi.
My new role does not require me to become a developer creating Apps. But I have been advocating and teaching fellow IT pro’s the importance of embracing developer tools and processes for our infrastructure management tasks. My knowledge around infrastructure as code is with PowerShell, Terraform, and ARM. My C# skills are very limited, although I have some experience.
Pulumi is definitely putting developers first, and I need to step up my game.
Azure Resource Manager and Azure Bicep are both domain-specific languages, meaning they only work with Azure. Terraform, is another popular tool (almost a standard), which also has it’s own language (HCL). HCL differs from ARM as it works with more than Azure.
Create, deploy, and manage infrastructure on any cloud using familiar programming languages and tools.
Pulumi
Pulumi on the other hand, use general-purpose programming languages. This means you can deploy and maintain your infrastructure with ‘real programming languages’, like C#, Java, TypeScript, and Go.
Pulumi is a declarative infrastructure as code tool. And it’s core engine will ‘build’ your desired infrastructure, and keep track of its state.
You start with something called a Project. The project folder is controlled via a Pulumi.yml file looking something like this, where name and runtime are mandatory.
name: core-infra
runtime: dotnet
description: my very first pulumi projectAfter creating the project you will need to create a stack. The stack is an instance of your project. For example, staging and production of project core-infra would be two separate stacks.
You might be familiar with this concept already, but if not here’s what’s what;
Pulumi keeps a snapshot of your infrastructure, referred to as ‘state’. This allows Pulumi to delete, create, and change your infrastructure components. But it also means you have to think about where you perform edits (only within the Pulumi stack/project), and where to store your state files.
By default Pulumi will store and manage state with their online service, Pulumi Console.
My short goal for self learning Pulumi is to replicate what I demoed in me and Marcel Zehner’s Live streams on Azure resource manager and infrastructure as code.
for Pulumi I am using this repository
For some reason, I assume you run Windows and CSharp, but if you fancy any of the other options, they are documented as well.
To run Pulumi on Azure you will need to install Pulumi, log in/sign up, install .NET 3.1, and Azure CLI (if you don’t have it already). The process is documented on the getting started page.
I tried to run with .NET 5.0, without any luck, but that might be solved soon.
Your next task is to create your project. In all essence, you run a few commands against an empty folder. This will generate the Pulumi program files and your project metadata files. Below is my configuration
cd C:users\MartinEhrnst\repos\Pulumi\
mkdir 1.ResourceGroup-storageAccount
cd 1.ResourceGroup-storageAccount
pulumi new azure-csharp
After filling in your mandatory project parameters, a getting started code will be generated for you. This will create an Azure resource group and a storage account.
In the above picture, I have changed this slightly to include a storage container, and change some of the default parameters. You can find my latest Pulumi code in this GitHub repo
For those experienced with C#, you can see that Pulumi has classes for the Azure resources. But since this is C#, we can use common coding techniques, like iterations (for-each) to deploy our infrastructure.
If I now want to deploy my infrastructure. I will need to run Pulumi, which translates this code into something Azure Resource Manager can understand. To my knowledge, Pulumi uses the Azure Resource Manager REST APIs to run the deployment.
To deploy the resources, you can follow this guide. In my environment above, this is the code and output from my review.
PS C:\Users\MartinEhrnst\repos\Pulumi\1.ResourceGroup-storageAccount> pulumi up
Previewing update (dev)
View Live:
Type Name Plan
+ pulumi:pulumi:Stack rg-and-storage-dev create
+ ├─ azure:core:ResourceGroup resourceGroup create
+ ├─ azure:storage:Account storage create
+ └─ azure:storage:Container container create
Resources:
+ 4 to create
Do you want to perform this update? details
+ pulumi:pulumi:Stack: (create)
[urn=urn:pulumi:dev::rg-and-storage::pulumi:pulumi:Stack::rg-and-storage-dev]
+ azure:core/resourceGroup:ResourceGroup: (create)
[urn=urn:pulumi:dev::rg-and-storage::azure:core/resourceGroup:ResourceGroup::resourceGroup]
[provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-]
location : "norwayeast"
name : "rg-PulumiStorage"
+ azure:storage/account:Account: (create)
[urn=urn:pulumi:dev::rg-and-storage::azure:storage/account:Account::storage]
[provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
accountKind : "StorageV2"
accountReplicationType: "LRS"
accountTier : "Standard"
allowBlobPublicAccess : false
enableHttpsTrafficOnly: true
isHnsEnabled : false
location : output<string>
minTlsVersion : "TLS1_0"
name : "storage2966fa9"
resourceGroupName : "rg-PulumiStorage"
+ azure:storage/container:Container: (create)
[urn=urn:pulumi:dev::rg-and-storage::azure:storage/container:Container::container]
[provider=urn:pulumi:dev::rg-and-storage::pulumi:providers:azure::default_3_33_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
containerAccessType: "private"
name : "images"
storageAccountName : "storageab46f04"
In Azure, I can now see that the storage account and resource group are created. But I cannot find this as deployments. I suspect this has to do with how Pulumi interacts with Azure resource manager. This might not be an issue for you, but if you rely on the deployment plane, you should have given this a thought.
Given my very limited knowledge of the product that is hard for me to answer. But there are things you should consider.
As I said, I have advocated for a few years about the ‘Modern IT pro’. Meaning we need to adopt and use more developer-oriented software and processes, like Git for example.
By using Pulumi you are not only adopting processes, but you also assume your team knows CSharp or any of the other supported languages. If your team consists of IT Pro’s who are beginning to explore the Dev side of the DevOps circle. Pulumi will give you some rough weeks ahead.
On the other hand, if your team is developer heavy, looking into the operations side. Pulumi might be your best choice. As a developer, it must seem alluring to be able to provision infrastructure together with your application code.
However, the responsibility for correct configuration, governance, and security is still the most important for your infrastructure. Can this be done with the same team and codebase, you can definitely consider using Pulumi.
A tool to convert ARM templates to Pulumi already exists. During my initial testing, I had success converting less complex templates, but when I tried to convert a nested template with a Copy loop the tool failed.
I suggest you try it out with your own templates, and since it’s open-sourced, you could always try to improve it your self. If not, the community will at some point.
Azure
For weeks Marcel Zehner and I have held four live streams. Covering ‘everything’ related to Infrastructure as code on Azure.
In the series, we covered the following topics, and everything is now available on YouTube