Azure Archives - Page 17 of 17

Weather Data in OMS Log Analytics

24/10/201610/04/2018
by Martin Ehrnst

If you’re one of the few who has seen my last blog posts about my SCOM weather management pack, you have probably figured that I am a bit too much in to the weather. Along side the weather management pack I looked in to getting the same type of data in to Microsoft Operations Management Suite (OMS) and the Log analytics part of it. I knew that OMS has a REST api that supports sending data without having to use any agents and i figured that’s perfect for my little weather study.

For around three weeks ago I took bits an pieces from my weather MP and made a powershell script that could output Json which is used in MSOMS API. On TechNet Brian Wren has written a guide on how to get started using the Data Collector API – I grabbet the already created functions and adapted those in to my script, placed it in Azure Automation and forgot the whole thing until last friday where i created a view for some of the data and posted it on Twitter

Community chief, Cameron Fuller reached out an told me he worked on the same thing. I contacted him by email and we exchanged our scripts and he shared some tips as well.

Enough with the history. The script we created have the ability to get weather data from yr.no (norwegian site) and openweathermap. Yr.no was what i used for SCOM, and OpenWeather was something Cameron was looking in to. There API’s are different, but OK to work with.

Setting the script together

The script has four functions two of them are from technet, and is required to get an autorization key, and the other one to send the data. These are well documented so i will go through the ones who get the data and how it ties together.

Get-YrWeatherData

YR.no has an XML based API. We get data from observations and the forcasted temperature. To use it you get the full URL from yr.no example: http://www.yr.no/place/Norge/Oslo/Oslo/Oslo/forecast.xml

#region YRno Config
$URLs = 'http://www.yr.no/place/Norge/Rogaland/Stavanger/Stavanger/forecast.xml', 
'http://www.yr.no/place/Norge/Hordaland/Bergen/Bergen/forecast.xml', 
'http://www.yr.no/place/norge/oslo/oslo/oslo/forecast.xml',
'http://www.yr.no/place/USA/New_York/New_York/forecast.xml',
'http://www.yr.no/place/Storbritannia/England/London/forecast.xml'
$YRLog = "YRno" #setting the log type for YRno
#endregion

param(
[Parameter(Mandatory = $false)]
[string[]]$locationURL
)
$LogType = "YRno" #setting the log type for YRno

if (!$locationURL){
$locationURL = 'http://yr.no/place/Norway/Oslo/Oslo/Oslo/forecast.xml'} #default URL to Oslo, Norway
#Create a table to accept multiple locations
$weatherTable = @()
foreach ($url in $locationurl){   
[xml]$yr = Invoke-WebRequest -Uri $URL -UseBasicParsing
[string]$locationName = $yr.weatherdata.location.name
#Getting the forcasted temperature
[int]$ForecastTemp = $yr.SelectNodes("//forecast").tabular.time.temperature.value | Select-Object -First 1
[int]$Forecastprecipitation = $yr.SelectNodes("//forecast").tabular.time.precipitation.value | Select-Object -First 1
[int]$observedtemp = $yr.SelectNodes("//observations").weatherstation.temperature.value | Select-Object -First 1
[string]$observedVindName = $yr.SelectNodes("//observations").weatherstation.windSpeed.name | Select-Object -First 1
[string]$observedVindDirectioName = $yr.SelectNodes("//observations").weatherstation.windDirection.name | Select-Object -First 1
#Output

$weatherData = @{
'LocationName' = $locationName
'ForecastedTemp' = $ForecastTemp
'Precipitation' = $Forecastprecipitation
'ObservedTemp' = $observedtemp
'WindDirection' = $observedVindDirectioName
'Wind' = $observedVindName
}

#add location weather data to our table
$weatherTable +=$weatherData

}
#Convert data to Json accepted by OMS
$weathertable  | ConvertTo-Json
}

Get-OpenWeatherMapData

Probably the one that is going to be used by the broad audience.

To use this you must sign up to OpenWeatherMap.org and obtain an API key. It is free for unless you use it for some commercial stuff or use huge amount of data.

The function uses a location ID inside the variable $Citys. I find it easiest to just grab it from the end of the location url after you have found your city. Paris FR, http://openweathermap.com/city/2988507
Chose between Imperial, Metric or Kelvin to adapt to your needs – who uses kelvin?

The current version has a bug where it only supports one location ID. We are looking in to it and will update when it’s fixed.

#region OpenWeathermap Config
$Citys = '3137115'
$Unit = 'Metric' #chose between Metric, Imperial or Kelvin
$OpenLog = "OpenWeather" #setting log type for OpenWeatherMap
#endregion

Param ($OpenWeatherMapKey, $Citys)
$LogType = "OpenWeather" #setting log type for OpenWeatherMap
$weatherTable = @()
Foreach ($city in $Citys){

$GetWeather = Invoke-RestMethod -uri "api.openweathermap.org/data/2.5/weather?id=$City&APPID=$OpenWeatherMapKey&units=$Units"
[String]$City = $GetWeather.name
[String]$WeatherDescription = $GetWeather.weather.description
[int]$Temp = $GetWeather.main.temp
[int]$WindSpeed = $GetWeather.wind.speed
[int]$BarometricPressure = $GetWeather.main.pressure
[int]$Humidity = $GetWeather.main.humidity

#Output

$weatherData = @{
'City' = $city
'Temp' = $Temp
'Humidity' = $Humidity
'WindSpeed' = $WindSpeed
'BarometricPressure' = $BarometricPressure 
'Description' = $WeatherDescription
}

$weatherTable += $weatherData
#Convert data to Json accepted by OMS
$weathertable  | ConvertTo-Json
}
#End Function
}

OpenWeather also have a good API documentation

Setting up Azure Automation part

We designed the whole thing to run in Azure automation and for it to be easy for others to use we utilize the ability to store encrypted variables to use inside your scripts.

Assuming you already have an azure automation account you go to: Automation accounts > ‘account’ >Assets and create the following variables

CustomerID
- This is the OMS workspace ID
SharedKey
- Primary key from your OMS workspace
OpenWeatherMapKey
- If using openweathermap. This is you api key

Finished, it should look like this

The next thing will be to create a azure automation runbook. I will suggest you use the ISE addon to create runbooks/workflows, but for this its a matter of copy and paste so web gui is fine. Below you will find the initial script release, but latest version is always available on GitHub

<#

    .DESCRIPTION
    OMS weather Solution - track weather forecast and observations within MSOMS

    Usage and Configuration:
    There are one config region per function. This script can get data from OpenweatherMap or Norwegian YR.no (not only norwegian locations)
    Edit each config area to fit your own environment.
    Script is intended to run in azure atuomation. You will have to create runbook assets to use this script
    If you want to run in another automation tool or on your own computer, please change the general variables

    In the end of the script. Comment out the function you do not want to use.

    .NOTES
    Version 1.5

    Martin Ehrnst /adatum.no /@ehrnst
    Cameron Fuller, Catapult systems /@cfullerMVP

    .CHANGELOG
    30.01.2017 v.1.5:
    Fixed multiple location issue for Open Weather Map.
    Thanks to 'jowildes' (blog comment) pointed out that there was some incorrect bracket placements causing the trouble
    Minor code changes

    October 2016 v1.1 
    (Initial release)

#>

#region General variables
$customerId = Get-AutomationVariable -Name 'CustomerID'
$SharedKey = Get-AutomationVariable -Name 'SharedKey'
$OpenWeatherMapKey = Get-AutomationVariable -Name 'OpenWeatherMapKey'
$time = [DATETIME]::Now
#endregion

#region OpenWeathermap Config
$Citys = '3137115', '2643743', '1880252' #Get your City ID from Open Weather Map URL
$Unit = 'Metric' #chose between Metric, Imperial or Kelvin
$OpenLog = "OpenWeather" #setting log type for OpenWeatherMap
#endregion

#region YRno Config
$URLs = 'http://www.yr.no/place/Norge/Rogaland/Stavanger/Stavanger/forecast.xml', 
'http://www.yr.no/place/Norge/Hordaland/Bergen/Bergen/forecast.xml', 
'http://www.yr.no/place/norge/oslo/oslo/oslo/forecast.xml',
'http://www.yr.no/place/USA/New_York/New_York/forecast.xml',
'http://www.yr.no/place/Storbritannia/England/London/forecast.xml'
$YRLog = "YRno" #setting the log type for YRno
#endregion

function Get-YrWeatherData{
<#
Get-YrWeatherData
uses yr.no xml api to get loaction forcasted and observed temperature.
Result is converted to Json and originally created for OMS data collector API

Version 1 September 2016
Martin Ehrnst /Adatum.no

NOTE: YR.no does not have observations for all locations.
#>


param(
    [Parameter(Mandatory = $false)]
    [string[]]$locationURL
)

if (!$locationURL){
    $locationURL = 'http://yr.no/place/Norway/Oslo/Oslo/Oslo/forecast.xml'} #default URL to Oslo, Norway
#Create a table to accept multiple locations
    $weatherTable = @()
foreach ($url in $locationurl){   
    [xml]$yr = Invoke-WebRequest -Uri $URL -UseBasicParsing
    [string]$locationName = $yr.weatherdata.location.name
#Getting the forcasted temperature
    [int]$ForecastTemp = $yr.SelectNodes("//forecast").tabular.time.temperature.value | Select-Object -First 1
    [int]$Forecastprecipitation = $yr.SelectNodes("//forecast").tabular.time.precipitation.value | Select-Object -First 1
    [int]$observedtemp = $yr.SelectNodes("//observations").weatherstation.temperature.value | Select-Object -First 1
    [string]$observedVindName = $yr.SelectNodes("//observations").weatherstation.windSpeed.name | Select-Object -First 1
    [string]$observedVindDirectioName = $yr.SelectNodes("//observations").weatherstation.windDirection.name | Select-Object -First 1

#Output

$weatherData = @{
    'LocationName' = $locationName
    'ForecastedTemp' = $ForecastTemp
    'Precipitation' = $Forecastprecipitation
    'ObservedTemp' = $observedtemp
    'WindDirection' = $observedVindDirectioName
    'Wind' = $observedVindName
    }

#add location weather data to our table
$weatherTable +=$weatherData

}
#Convert data to Json accepted by OMS
$weathertable  | ConvertTo-Json
}

Function Get-OpenWeatherMapData {

<#
Get-OpenWeatherMapData

Uses openweathermap.com api to get weather data and inserts in to OMS log analytics
Version 1.0 January 2017
Created by Cameron Fuller & Martin Ehrnst

#>

Param ($OpenWeatherMapKey, $Citys)
$weatherTable = @()
Foreach ($city in $Citys){

    $GetWeather = Invoke-RestMethod -uri "api.openweathermap.org/data/2.5/weather?id=$City&APPID=$OpenWeatherMapKey&units=$Unit"

    [String]$City = $GetWeather.name
    [String]$WeatherDescription = $GetWeather.weather.description
    [int]$Temp = $GetWeather.main.temp
    [int]$WindSpeed = $GetWeather.wind.speed
    [int]$BarometricPressure = $GetWeather.main.pressure
    [int]$Humidity = $GetWeather.main.humidity

    #Output
    $weatherData = @{
    'City' = $city
    'Temp' = $Temp
    'Humidity' = $Humidity
    'WindSpeed' = $WindSpeed
    'BarometricPressure' = $BarometricPressure 
    'Description' = $WeatherDescription
    }

    #add location weather data to our table
    $weatherTable +=$weatherData
    }
    #Convert data to Json accepted by OMS
    $weathertable  | ConvertTo-Json
}
#End Function
# Function to create the authorization signature - TECHNET example
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
  $xHeaders = 'x-ms-date:' + $date
  $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

  $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
  $keyBytes = [Convert]::FromBase64String($sharedKey)

  $sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
  $sha256.Key = $keyBytes
  $calculatedHash = $sha256.ComputeHash($bytesToHash)
  $encodedHash = [Convert]::ToBase64String($calculatedHash)
  $authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
  return $authorization
}

#Send data to OMS - a technet example
Function Send-OMSData($customerId, $sharedKey, $body, $logType)
{
  $method = 'POST'
  $contentType = 'application/json'
  $resource = '/api/logs'
  $rfc1123date = [DateTime]::UtcNow.ToString('r')
  $contentLength = $body.Length
  $signature = New-Signature `
  -customerId $customerId `
  -sharedKey $sharedKey `
  -date $rfc1123date `
  -contentLength $contentLength `
  -fileName $fileName `
  -method $method `
  -contentType $contentType `
  -resource $resource
  $uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'

  $headers = @{
    'Authorization'      = $signature
    'Log-Type'           = $logType
    'x-ms-date'          = $rfc1123date
    'time-generated-field' = $time
  }

  $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
  return $response.StatusCode
}


$YRdata = Get-YrWeatherData -locationURL $URLs
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($YRdata)) -logType $YRlog
$YRdata


$Opendata = Get-OpenWeatherMapData -OpenWeatherMapKey $OpenWeatherMapKey -Citys $Citys
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($Opendata)) -logType $OpenLog
$Opendata

After the script is in azure, please run a test to see if everything is alright

When everything is functioning correctly, add a schedule to the runbook and wait until tomorrow. you should have some cool data points to work with

When searching for your data. Remember dat OMS adds a default suffix “_CL” to the end of all custom data types. Fields are also getting an “_s” for string etc. You can see all custom fields from the configuration area in OMS

Time to start to play with your data

Typing Type=OpenWeather_CL | measure avg(ObservedTemp_d) by City_s interval 1hour in to your search will give a time chart similar to this.

Now, weather data is just an example, but whit the ability to send data through OMS data collector API and create our own solutions/dashboards inside OMS i know we will see some cool stuff in a short time.

Automation

Using webhook in scom subscription (POC)

09/09/201611/09/2016
by Martin Ehrnst

Recently, I was at System Center Universe (Now, Experts Live) in Berlin. During one side session i got talk to Cameron Fuller (Microsoft MVP) and I asked if he knew if anybody had used azure automation and it’s webhook capabilites in a SCOM subscription.

At Intility (my employer) we have developed an alert central which handles alerts from Solarwinds and other systems we use daily. OpsMgr is one of few systems that still tap directly in to our ticketing system, but were working on a change here. Webhooks and azure automation is one thing were looking at.

My attempt (if possible), using webhooks and Azure automation is to be able to send alerts from SCOM, and not have something query for new alerts when there nothing to handle. Cameron quickly reached out to his fellow MVPs and a few ours later i got a reply. Turns out that Jakob Gottlieb Svendsen (MVP) from Coretech had used webhooks in a recovery task. He basicly done all the work now :).

http://blog.coretech.dk/jgs/azure-automation-scom-triggering-a-runbook-as-a-recovery-task-using-webhooks/ (read this :))

The rest of this post will quickly show how to.

Create azure automation account
Create a new runbook
Add webhook
Runbook parameter
SCOM Command Channel and subscription

Azure Automation Account

Search for automation accounts in services and add a new Automation account. Use an existing resource group or create a new. After wizard is finished we should have an account assigned to a subscription.

By now you have probably read Jakobs blog and you have a new runbook in place, but i will guide you through anyway 🙂

Create a new runbook and use “Powershell workflow”. Once finished you will be sent to the online editor which is fine for now, but i suggest you download azure automation addon for ISE if you plan to do a lot of scripting here 🙂

Azure Automation PS worflow will start like this. If you add script parameter and publish where able to create a webhook. Place your normal PS script between the two curly brackets.

workflow 'your runbook name'

{

Param [object]($WebhookData)

}

I have adde the parameter already, that way we are able to create webhook an parameter at the next step.

Next task is to create a webhook for your runbook. With a webhook you are able to call the runbook with a URL and also pass parameters with JSON data inside the url. (Heres where the parameter for your webhook and script comes in)

Remember, copy your webhook URL – it’s not available after it is created

After youre ‘hooked’ add paramters and define where to run. For this test i just fire it off in azure, but you can run your script on prem with the hybrid worker. Use your script parameter in the webhook data input.

You have probably added your own script inside the Azure workflow already, but here is my OpsMgrAlertHandlerTEST script which is based on Jakobs example and will output some key alert datato the console. Instead of just output it to the console i can now pass this information to our alert central by triggering a new runbook or adding it directly within this script.

workflow OpsMgrAlertHandlerTEST

{

param (

[object]$WebhookData

)



    $WebhookName    =   $WebhookData.WebhookName

    $WebhookHeaders =   $WebhookData.RequestHeader

    $WebhookBody    =   $WebhookData.RequestBody



    $Inputs = ConvertFrom-JSON $webhookdata.RequestBody

    $ComputerName = $Inputs.ComputerName

$AlertName = $Inputs.Alert

$ResolutionState = $Inputs.State

$AlertID = $Inputs.AlertID



    Write-Output "Computername: $ComputerName"

Write-Output "Alert: $AlertName"

Write-Output "State: $ResolutionState"

Write-Output "ID: $AlertID"

Write-Output "$Inputs"

}

From scom we will use powershell to POST alert data with invoke-restmethod and the URL you copied from the webhook configuration, right?

Again Jakob is apparently our JSON webhook guy http://blog.coretech.dk/jgs/azure-automation-using-webhooks-part-1-input-data/

SCOM Command Channel and subscription

Our final step is to create a command channel, subscriber and a subscription to trigger the runbook. From the admin panel in scom create a new command notification channel

In settings we will add path to Powershell and a command to run. Remember. Webhook url and the parameters to POST

Here is my full commandline (i have removed my webhook uri here)

-executionpolicy Unrestricted -Command " &{Invoke-RestMethod -Method Post -Uri 'YOUR WEBHOOK URL' -Body (ConvertTo-Json -InputObject @{'ComputerName'='$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$';'Alert'='$Data[Default='Not Present']/Context/DataItem/AlertName$';'State'='$Data[Default='Not Present']/Context/DataItem/ResolutionStateName$';'AlertID'='$Data/Context/DataItem/AlertId$'}) -ErrorAction Stop}"

Inside the JSON string we will pass Computer Name, Alert name and the resolution state. Modify this by using the picker on the right side. After you have created the channel, create a new subscriber to use with this channel and finally create the actual subscription

For this test/POC i have chosen to send all information alerts to this subscription. These are mostly rules i can close (and generate) during testing.

Subscriber and channel are the two we just created.

At this point you should have:

Azure automation account
Automation runbook, my example or your own
Web hook enabled
Command channel
Subscriber and subscription

Finally, lets test our solution. I have added all informational messages to this subscription. I will close one of those and cross my fingers. We can follow the process in azure portal

As you see the job is queued and in the output console we see the output from our webhook data

It worked!

As this is totally in a proof of concept state for us i would greatly apriciate inputs on how we can accomplish the task using other methods.

As we see, using webhooks in a subscription works quite well. But i havent implemented it in our production environment or sent it to our alert central yet. I see some issues, one being limited by how many powershell scripts we can run at once. Jakob suggested maybe we will look in to creating a connector and let us query a subscription for new alerts but i haven’t gotten around to try it out.