Automation Archives

Using Azure OpenAI ChatGPT for incidents

24/04/202324/04/2023
by Martin Ehrnst

The Challenge of Long Chat Threads During Incidents

Everyone is an AI expert these days. After OpenAI released ChatGPT it’s almost impossible to work in tech without being presented with good (and bad) examples of what this service can do for you. Of course, I am also interested in seeing what these services can help with, but until very recently I didn’t have any good use cases that would provide the company I work for and me any value. Of course, GitHub copilot help our developers when it comes to programming, and it helps me from time to time with the same. Maybe even more, as I have no idea what I am doing in most cases.

I used Azure OpenAI and the GPT3.5 turbo language model, which is what the public ChatGPT uses. The reason for using Azure OpenAI is to make sure we can use and train the model with our company-specific data. Currently, the service is open on request only.

Anyway, let’s dive into the actual problem at hand. As an incident responder, you know how important it is to keep track of what happened during an incident. When the incident is over, you need to have a clear summary of what happened to help you identify what went wrong and how you can prevent similar incidents in the future. But what if you have hundreds or even thousands of chat messages to go through? That’s where Azure OpenAI and ChatGPT come in.

Using PowerShell and Slack API to Retrieve Chat Threads from Slack

In Vipps Slack is our primary communication tool during incidents. After each incident, we need to go through the chat threads to understand what happened. We used to do this manually, but it was time-consuming and error-prone. That’s why we decided to automate the process using Azure OpenAI and ChatGPT.

First, I used PowerShell and Slack API to retrieve the individual messages from Slack threads. I then passed the chat threads to Azure OpenAI’s GPT-3 model, which generated a summary of the incident based on the chat messages.

Setting up a Slack App to use Slack API

To use the Slack API to retrieve chat threads, you’ll need to set up a Slack app and obtain an API token. Here’s how you can do it:

Go to the Slack API website (https://api.slack.com) and sign in with your Slack account.
Click on the “Create New App” button and give your app a name and a development workspace.
In the app dashboard, navigate to the “OAuth & Permissions” section and add the “channels:history” scope to your bot token scopes.
Install the app in your workspace and copy the Bot User OAuth Access Token.
In PowerShell, use the following code to retrieve the chat threads from Slack:

$token = "YOUR_BOT_TOKEN"
$channelId = "CHANNEL_ID"
$url = "https://slack.com/api/conversations.history?token=$token&channel=$channelId"
$response = Invoke-RestMethod -Uri $url

Output from the Azure OpenAI PowerShell script

Putting the Complete PowerShell Script together

Below is a complete PowerShell script. I have redacted some company-specific information. So feel free to use and modify where you need. It is possible that you need to filter out more things than I did. Specific users that you do not want to include, HTML content, etc.

	# slack test

	$slackKey = Get-AzKeyVaultsecret –VaultName "" –Name "" –AsPlainText
	$azOpenAiKey = Get-AzKeyVaultsecret –VaultName "" –Name "" –AsPlainText
	$slackChannelId = ""
	$slackThreadId = ""
	$openAiUrl = ""
	$slackUrl = "https://slack.com/api/conversations.replies?channel=$slackChannelId&ts=$slackThreadId&pretty=1"
	$slackHeaders= @{
	"Authorization" = "Bearer $slackKey"
	"content-type" = "authorization/x-www-form-urlencoded"
	}

	$incidentThread = Invoke-RestMethod –uri $slackUrl –Method Get –Headers $slackHeaders


	$incidentMessages = ""
	foreach ($message in $incidentThread.messages) {
	$messageText = $message.text
	$messageUser = $message.user
	$messageUserUrl = "https://slack.com/api/users.info?user=$messageUser&pretty=1"
	$messageUser = Invoke-RestMethod –uri $messageUserUrl –Method Get –Headers $slackHeaders
	$messageUser = $messageUser.user.profile.real_name_normalized
	$messageDate = (([System.DateTimeOffset]::FromUnixTimeSeconds($message.ts)).DateTime).ToString("yyyy-MM-dd HH:mm")
	$messageText = $messageText.Replace("`n", "")
	$messageText = $messageText.Replace("`r", "")
	$messageText = $messageText.Replace(">", "")
	$messageText = $messageText.Split("https://")[0]

	$incidentMessages+="$messageDate\|$messageUser\|$messageText\|`n"
	}

	$openAiHeaders = @{
	"Content-Type" = "application/json"
	"api-key" = $azOpenAiKey
	"accept" = "application/json, text/plain, /"
	"accept-language" = "en"
	}

	$messages = $incidentMessages \|Out-String
	$prompt = @"
	summarize the incident chat log in the following format returning a structured markdown document. Make sure to include the following fields as headings
	Start time:
	End time:
	Observed issue:
	Root cause:
	Action items:
	Participants:
	$messages
	summary:
	"@

	$body = @{
	"prompt" = "$prompt"
	"max_tokens" = 500
	"temperature" = 0.3
	"top_p" = 1
	"frequency_penalty" = 0.2
	"presence_penalty" = 0.2
	"stop" = "tokens"
	} \| ConvertTo-Json –EscapeHandling EscapeNonAscii

	$summary = Invoke-RestMethod –uri $openAiUrl –Method Post –Headers $openAiHeaders –Body $body
	$summary.choices.text

view raw azopenai-slack.ps1 hosted with ❤ by GitHub

Summary and things to know

When using Azure OpenAI and ChatGPT to generate incident summaries, it’s important to keep in mind that the model is designed to analyze text-based data. If the chat messages contain images or other non-text data, the model will not be able to interpret them.

If your incident chat threads contain images or other non-text data, you may need to consider alternative ways to include that information

Another important consideration when using Azure OpenAI and ChatGPT to summarize chat threads from Slack during incidents is the token limitations of the model. The GPT-3 model has a limit of 2048 tokens per input, which means that if your chat threads are particularly long, you may need to split them into multiple inputs to generate a complete summary.

In addition, the Azure OpenAI API uses a token-based pricing model, which means that you will be charged based on the number of tokens generated by the model. If you’re generating a large number of summaries or working with particularly long chat threads, this can quickly become a significant expense.

By being mindful of these token limitations and experimenting with different summarization strategies, you can still use Azure OpenAI and ChatGPT to generate valuable incident summaries that can help you improve your incident response process.

Image from chatgpt showing i used chatgpt to write the post

Automation

Remediate Azure Policy with PowerShell

11/06/202011/06/2020
by Martin Ehrnst

Azure Policy is there to help us with properly governed and secure infrastructure. However, Azure Policy requires management as well.

Lately, I have built a new set of policies to ensure diagnostic logs are forwarded to Azure Monitor Logs. Multiple policies and a policy initiative were deployed to multiple subscriptions and multiple customers. All this was made possible since we manage through Azure Lighthouse.

Automatic remediation of Azure Policy

The challenge faced after deploying the policy was how to remediate them. Since policies with effect ‘deployIfNotExists’ only apply to new or modified resources, I faced the job with clicking in the portal or figure out a way to do this with PowerShell.
I actually started with the portal, as I thought it would be a quick job. After doing one or two subscriptions I realized how much time I would use.

Given the fact that the imitative it self contained around 50 individual policies, and at the time 19 subscriptions. I figured spending some time in PowerShell was well worth it. There is also a pretty good chance I will find my self in the same situation pretty soon.

Create remediation task with PowerShell

To create a remediation task for a policy set you can use this script. It will connect to your subscription and get all non-compliant policies. Then start a policy remediation task for the individual policies.

Summary

Policies with effect “deployIfNotExist” only work for resources that are updated or created after the policy was applied. To remediate existing resources you will have to create the remediation tasks manually through the portal, or by using PowerShell (and REST API)

By the way, fellow Azure MVP Tao Yang has created everything you need in order to enable these policies your self. Please see GitHub for complete ARM templates. And please help him maintain everything by contributing.

Automation

Multi subscription deployment with DevOps and Azure Lighthouse

11/03/202003/09/2020
by Martin Ehrnst

Companies today are rapidly adopting new technology. Adding more pressure on the companies IT-department or the service provider. No matter where the workload runs, governance, security, and deployments are fundamental parts.
Azure Lighthouse came last year, and while it solves a lot of our trouble. Multi subscription deployments aren’t one of them – but it sure makes it more possible!

Azure deployments with Azure DevOps

One of the things that make service providers great at what they do is standardization. For example; making sure all subscriptions (customers in many cases) have the same set of baseline Azure policies.

Azure DevOps has multiple ways to deploy resources in Azure or other places. How do we deploy the same Azure template to multiple subscriptions using the same pipeline? In our case, I solved this with some existing features, forward-thinking and PowerShell magic.

Azure DevOps service connection with Azure Lighthouse

Multi subscription deployments with Azure DevOps is not a built-in feature. With Azure Lighthouse it became a little bit easier but will require some work.

First, you must set up a service connection and allow that to access one of your internal subscriptions. In Azure DevOps service connections are bound to one subscription.

For this service connection to be capable of multi subscription deployments, it will need access to your customer’s subscriptions. This can be solved through delegated resource management and Azure Lighthouse.
In my case, I had a group with contributor access. And I could add the SPN to that group. Otherwise, you will have to update your current delegation.

Repository structure

Everything you need, including a YAML pipeline is available on GitHub, but I will walk you through how and why I set it up.

I needed to create not only a solution to deploy one resource to multiple subscriptions. I also needed to deploy multiple ARM templates.

For this purpose, I had the option to create one large PowerShell script with complex logic. Or, I could reuse the same script for every deployment. I chose option two. My code repository now looks something like this

ARM-Templates (folder)
- storage-Account (folder)
  - azuredeploy.json
  - azuredeploy.parameters.json
  - deploy.ps1
- another-resource (folder)
  - azuredeploy.json
  - azuredeploy.parameters.json
  - deploy.ps1
- […]
azure-pipelines.yaml

PowerShell magic?

In regular deployments, we can use built-in tasks in the pipeline and deploy directly. For multi subscription deployment, PowerShell is my weapon of choice.

To people with PowerShell competency. The script used is fairly simple;

Connect to Azure
Retrieve the subscriptions
Iterate and deploy the ARM template(s) to each customer subscriptions.

Below is a short example. Showing the core in my deployment script (deploy.ps1)

$deploymentName = "Multi-sub-deployment"
$deploymentLocation = "westeurope"
$templateFile = ".ARM-Templatesstorage-Accountazuredeploy.json"
$templateParameterFile = ".ARM-Templatesstorage-Accountazuredeploy.parameters.json"

# getting all subscriptions
$subscriptions = Get-AzSubscription | Where-Object { $_.Id -NotIn $excludedSubs }

foreach ($subscription in $subscriptions) {
        
    # set context to the current subscription
    $subscriptionId = $subscription.id
    Set-AzContext -SubscriptionId $subscriptionId

    # deploy the arm template
    New-AzSubscriptionDeployment -Name $deploymentName -Location $deploymentLocation `
        -TemplateParameterFile $templateParameterFile -TemplateFile $templateFile
}

Multi subscription deployment, build pipeline

Although my repository contains a YAML pipeline, you don’t have to use it. To be honest, I’m not sure I like them. I used too much time trying to wrap my head around it. And at this point, it seems unfinished from the Azure DevOps side. Therefore, I will show you how to set up your pipeline to support multi subscription deployments using the classic method.

kick off with the classic mode, and chose the empty job on the template page. We could probably discuss if we even need multiple pipelines for this. But it doesn’t hurt, and it will be easier for the next person if we do this by the book.

For the build pipeline, I am renaming my job to something meaningful, and add one single task. Publish pipeline artifact

After you save and run the build. An artifact should be produced. The result should look something like this

Multi subscription deployment, release pipeline

After a successful build (or in this case copy files), it is time to create our release pipeline. When using YAML, the two pipelines are combined. Not at all confusing for someone like me, who not that many months ago, didn’t know anything at all about this stuff.

Once again, I start off with an empty job, before I add my artifact from my build pipeline. I also renamed the first stage to “resource deployment”.

Now it’s the matter of adding tasks to our job, and it’s here you will need the service connection that you added earlier. The task we are working with is the Azure PowerShell task. And for multi subscription deployment, it is the only task you’ll need. Below is my task configuration

For some reason, it takes a few tries before pipelines want to work. It might be because of lat hours, or a law created by some guy named Murphy. Anyway, once it’s up and running you shole be able to see something similar to my output below;

2020-03-10T17:13:12.2503976Z ## Az module initialization Complete
2020-03-10T17:13:12.2517976Z ## Beginning Script Execution
2020-03-10T17:13:12.2532801Z Generating script.
2020-03-10T17:13:12.3293381Z ========================== Starting Command Output ===========================
2020-03-10T17:13:12.3416275Z ##[command]"C:Program FilesPowerShell6pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'd:a_temp112fe6df-8ee0-4a10-b53e-5694a4d34b0f.ps1'"
2020-03-10T17:13:25.0914923Z No subscription specified. Deploying to all subscriptions
2020-03-10T17:13:25.2151812Z 
2020-03-10T17:13:25.2211374Z Name                                     Account             SubscriptionName    Environment         TenantId
2020-03-10T17:13:25.2292766Z ----                                     -------             ----------------    -----------         --------
2020-03-10T17:13:25.2300155Z MVP-Sponsorship (6dca9329-fb22-46cb-826…  MVP-Sponsorship     AzureCloud          22046864-98a9-4a9…
2020-03-10T17:14:01.7100820Z 
2020-03-10T17:14:01.7151741Z Id                      : /subscriptions/6dca9329-fb22-46cb-826c-/providers/Microsoft.Resources/deployments
2020-03-10T17:14:01.7152947Z                           /Multi-sub-deployment
2020-03-10T17:14:01.7154721Z Location                : westeurope
2020-03-10T17:14:01.7159918Z ManagementGroupId       : 
2020-03-10T17:14:01.7161790Z ResourceGroupName       : 
2020-03-10T17:14:01.7162557Z OnErrorDeployment       : 
2020-03-10T17:14:01.7163149Z DeploymentName          : Multi-sub-deployment
2020-03-10T17:14:01.7163805Z CorrelationId           : d49c10f8-0260-49d5-aa8d-08a41591d1a7
2020-03-10T17:14:01.7164463Z ProvisioningState       : Succeeded
2020-03-10T17:14:01.7165107Z Timestamp               : 3/10/2020 5:14:00 PM
2020-03-10T17:14:01.7166059Z Mode                    : Incremental
2020-03-10T17:14:01.7166610Z TemplateLink            : 
2020-03-10T17:14:01.7167216Z TemplateLinkString      : 
2020-03-10T17:14:01.7167793Z DeploymentDebugLogLevel : 
2020-03-10T17:14:01.7168836Z Parameters              : {[rgName, Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable], 
2020-03-10T17:14:01.7169800Z                           [location, Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable], 
2020-03-10T17:14:01.7173723Z                           [storagePrefix, 
2020-03-10T17:14:01.7174379Z                           Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.DeploymentVariable]}
2020-03-10T17:14:01.7175014Z ParametersString        : 
2020-03-10T17:14:01.7177870Z                           Name             Type                       Value     
2020-03-10T17:14:01.7178513Z                           ===============  =========================  ==========
2020-03-10T17:14:01.7179228Z                           rgName           String                     adatum    
2020-03-10T17:14:01.7179831Z                           location         String                     westeurope
2020-03-10T17:14:01.7180452Z                           storagePrefix    String                     str       
2020-03-10T17:14:01.7181006Z                           
2020-03-10T17:14:01.7181454Z Outputs                 : {}
2020-03-10T17:14:01.7181916Z OutputsString           : 
2020-03-10T17:14:01.7182291Z 
2020-03-10T17:14:02.0220641Z 
2020-03-10T17:14:02.0937716Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop

If you look at the output, you can see that the script set’s context to a subscription, and that there is no subscription specified in the pipeline.

Multi subscription deployment summary

Multi subscription deployments with Azure DevOps is not available as a default. But with a little bit of PowerShell trickery, you got a great solution. For service providers and large enterprises, Azure Lighthouse is now a preferred way to manage resources.

By granting a service principal access to your customer subscriptions (or internal for that matter), and use this SPN as a service connection in your Azure Pipeline. You can use PowerShell to iterate through each subscription and deploy the resources needed.

The beauty with this is that it will work regardless of where your DevOps environment is hosted. You can have separate tenants for Lighthouse, Azure DevOps and workplace.

This post described the following, which is required for multi subscription deployment to work.

Created an SPN in the management tenant
Authorized the service principal through Azure Lighthouse
Created a repository and added our scripts and templates to it
Created pipelines and used the SPN as our service connection
Used PowerShell and the built-in task to iterate through each subscription and perform deployments.

Using Azure OpenAI ChatGPT for incidents

The Challenge of Long Chat Threads During Incidents

Using PowerShell and Slack API to Retrieve Chat Threads from Slack

Setting up a Slack App to use Slack API

Putting the Complete PowerShell Script together

Summary and things to know

Share this:

Remediate Azure Policy with PowerShell

Automatic remediation of Azure Policy

Create remediation task with PowerShell

Summary

Share this:

Multi subscription deployment with DevOps and Azure Lighthouse

Azure deployments with Azure DevOps

Azure DevOps service connection with Azure Lighthouse

Repository structure

PowerShell magic?

Multi subscription deployment, build pipeline

Multi subscription deployment, release pipeline

Multi subscription deployment summary

Share this: