Azure Automation with Azure Active Directory Authentication

I finally got a bit of time and an excuse to test the integration of Azure Automation with Azure Active Directory authentication. This topic is very important to me for two reasons – firstly because I’m a big advocate of automation and I found the setup of automation, with certificate authentication, cumbersome and therefore a barrier of entry and secondly because using AD will tie automation very nicely to the recently announced role-based access control for Azure

So – to get started, I’ve created a use specifically for automation in the directory used to control access to my subscription. at this point in time Automation will only work with ‘organisational uses’ (i.e not a Microsoft-Account user) but that’s perfectly fine for me.

The User was created as a normal ‘User’, it did not need to be an administrator of any type.

Next, I’ve added the user as a co-admin of the subscription using the Azure Management Portal.

Following that I signed-in to the management portal with that user which gave me the opportunity to change the temporary password to a permanent one.

With the user configured and ready I entered my automation account, switched to the the Assets tab and clicked the Add Setting button on the tool bar. In the dialogue that opened I selected ‘Add Credential’

image

I then selected Windows Powershell Credentials as the Credential Type and entered a friendly name and description

image

Finally – I entered the username (which is the email address within the directory’s domain) and password

image

With the credentials all set-up I went on to write a simple workbook – I wanted to start all the VMs in my subscription, and so my ‘stop’ workbook looks like this –

workflow StartVMs
{
    $cred = Get-AutomationPSCredential -Name "Automation User"
    
    Add-AzureAccount -Credential $cred
    
    Select-AzureSubscription -SubscriptionName "<subscription name here>"
    
    InlineScript 
    {
        $listVM = Get-AzureVM
            foreach ($vm in $listVM) 
                { 
                    Start-AzureVM -ServiceName $vm.ServiceName -Name $vm.Name
                }
    }
}

One of the mistakes I did initially is to specify the Azure Active Directory username when calling Get-AutomationPSCredential instead of the friendly name I assigned when creating the Credential asset which resulted in an authentication error.

Programmatic access to Azure usage data

An exciting announcement, on a boring sort of topic, landed on my mailbox just now – finally the Azure usage details, so far only available as a download through the Azure Enterprise Portal, has been made available via a REST API (its documentation is available here although somewhat unfortunately this requires access rights to the portal)

The API is very simple, and at this point only allows retrieving the list of available months as a JSON document, the usage summary for selected month, also as a JSON document and the detailed usage for a selected month as a CSV file.

Sure – this could be improved on, but it is the most basic functionality that so many customers have been asking for – the ability to programmatically, and therefore regularly, download the usage reports so that they can be reported on (and, where necessary, actioned on)

The documentation include sample code in C# and powershell which is very clear and ‘just works’, so there’s no point me repeating it here. it is a simple case of submitting an HTTP Request to a properly constructed URL and adding a bearer HTTP token with an access key generated through the EA portal itself and processing the JSON or CSV result, depending on the request issued.

I can see most organisations building either an SSIS job or similar, that downloads the data daily (it refreshes every 24 hours) and loads it into a database from which reports and alerts can be driven using existing BI capabilities or simply downloading the CSV file and storing it in a central location such as a SharePoint library to be analysed by Excel and PowerView.

Eitherway – having programmatic access to this data in a way that allows plugging it in to exiting BI practices is a powerfull thing and a well sought after capability.

Using AD for authentication between Web app and Web API

In my earlier post about the problem with ACS I promised to blog about how to configure Microsoft Azure Active Directory (MaaD) to support a client app calling a Web API using a client secret (as opposed to presenting an authentication dialogue to an interactive user).

To get started (and save me a lot of typing) I suggest you get started by reading Johan Danforth’s great blog post on how to do this for an interactive login as this is where I started)

If you follow the instructions carefully you will have a Web API published and registered with AD and you will have updated the application’s manifest file in AD to allow you to configure permissions to it in other applications in the directory)

You will also have a client app, also configured in AD (as a ‘native application’) in which you have configured permission to access the Web API and the relevant call to issue the request to AD to get the token and to the Web API bearing that token.

Crucially – you will have written very little code to do this. AD, the ADAL Nuget Package and the OWIN library (on the server side) will have done all the heavy lifting for you.

What I wanted to do is to remove the interactive user from the equation – in my scenario a user uses a web app, which in turn uses my Web API, but this is not a delegation scenario, the web app can call the web api on its own right.

To achieve this, I needed to make two changes from Johan’s example –

Firstly I needed to register my web app as a client app which isn’t a native client as a native client app in AD does not have configuration for client secret. so I’ve added a new app to my directory, made sure to select the ‘web application and/or web api application’ for the type and configured a name, APP ID and (in my case a mock) sign-on url.

I had also set the permission from my web client application to the web-api, in the same way that I did earlier for my native client.

And finally – I added a new key by selecting a duration and copied the secret (important: the key cannot be accessed after its creation, make sure to copy it)

With the AD configuration done I make the relevant changes to the client code to use the client secret rather than the interactive sign on

The original code includes the following line - 

var ar1 = ac.AcquireToken(“https://. . . WEB Api APP ID URI. . .”,
              "e9b5d821-. . .Native Client’s Client Id",
              new Uri("http:// . .  Native Client’s APP ID URI"));

so support authenticating using client secret, this needed to change to

ClientCredential cc = new ClientCredential("e9b5d821-Web Client Client ID",
"Qxc49i. . .client secret generated in AD portal"); var ar1 = ac.AcquireToken(https://. . . WEB Api APP ID URI. . .”, cc);

and with this done (the rest of the code remains unchanged), when I now run the client, rather than being prompt to authenticate the call to the web api is made and authentication is done using the client secret.

In closing, for completeness, the interaction really translates to two web requests – the first is to AD to retrieve the access token, and in my case this looks like this –

POST https://login.windows.net/[1db093b3-…AD tenant ID]/oauth2/token HTTP/1.1

Host: login.windows.net

Content-Type: application/x-www-form-urlencoded

Content-Length: 212

grant_type=client_credentials&client_id=[e9b5d821-…client app client id]&client_secret=[Qxc49iXn…client secret]&resource=[https%3A%2F%2F…..protected resource app ID]

This request, if successful, returns a JSON response which includes a JWT token from AD that can be used in subsequent requests to the application. That subsequent requests looks like this (token trimmed for readability) –

GET http://localhost/WebApplication1/api/Values HTTP/1.1

Host: localhost

Authorization: Bearer eyJ0eXAiOiJKV1QiLCJh……..

Finally – whilst I was experimenting with this I encountered a few errors that are worth noting here –

The one I was most confused by was

"AADSTS90014: The request body must contain the following parameter: ‘client_secret or client_assertion’."

In my case this happened when I tried to request a token from AD through my client app when it was still coded for interactive user sign-on. when the AD application used was native, using the interactive logon code worked fine, as soon as I tried to use a web api AD application without changing the code to use client_secret I got the error above.-

Final thought – ACS is slowly being embedded in AD but for the time being I find the experience a little bit fiddly and short of documentation (in particular the Manifest file manipulation) and lack of flexibility

The integration above currently only works with JWT tokens and client secret authentication (SWT is not supported I believe as is certificate based authentication) but this will be corrected over time, I’m sure.

 

[cross posted on the Solidsoft blog]

The problem with ACS….

…is that it currently has no published SLAs and there’s no out-of-the-box solution for Disaster Recovery in an active scenario (as in – web services, REST services)

Let me expand on these two points –

I could not find any SLA information about ACS, so I asked Microsoft who responded that, as things stand ACS is a free service and they do not offer SLAs for free. This might change as and when ACS is properly rolled into Microsoft Azure Active Directory with its paid, Premium option, but that’s not the case at the moment.

Pragmatically – I have not seen any ACS issues in the past and there are several Azure services that do carry SLA that take a dependency on ACS, namely the Service Bus and BizTalk Services. Make of that what you will – you have to either accept or reject this risk.

To make matters worse – there’s no easy solution for DR for ACS in an active scenario.

Federated identity follows two approaches – when dealing with users and browsers the passive approach is used and that relies on browser redirects. This means that the web app is responsible for redirecting users to the Identity Provider (IdP) and holds the configuration of where the IdP is. That means, for example, a web role deployed across data centres and fronted by Traffic Manager, can have separate configurations to redirect users to different ACS namespaces, depending on the environment.

When dealing with services (SOAP with WS-Federation or REST with oAuth) the service client holds the configuration about where the IdP is (in the SOAP case this is part of the service contract in oAuth it is simply exchanged out of band) and it is the service client’s responsibility to go and get a token from the IdP before calling the service.  This means that if the ACS namespace changes (for example due to outage and switch to DR), the client will need to know to go to the alternative the service won’t tell it automatically.

With this in mind, what are the options?

I can think of 3 or 4. would love to hear additional views –

1. Accept the risk and the need to reconfigure clients when switching to DR.

2. If you have control over the client – implement an operation to automatically (on failure) and/or regularly request the metadata from your web role, which would allow it to auto-recover

3. If either of the above are not acceptable – roll your own oAuth solution on your web role which would mean that client access the same url (managed by traffic manager) to access the service and the IdP.

A better alternative is to use the bit of ACS that is rolled into Active Directory through the configuration of Applications for both the Web API and the client. I’ll post an entry on this shortly. the upside is that if you then make sure your directory is a Premium one you are covered by SLAs, although I’m still not sure what the DR story for AD is.

The main downside of this approach at this point in time is that, currently at least, there’s not much configuration that can be done – it only supports a client secret and JWT tokens with no control over token TTL etc. I also the the process is quite fiddly, requiring tempering with the app manifest manually etc, but I’m sure all of this will improve over time.

[cross posted on the Solidsoft Blog]

Caching choices

In a meeting with a customer this week the question of which caching capability on Azure to use came up again.

As I’m writing this Microsoft Azure offers three caching options, here’s the extract from the Azure web page –

Azure Cache is a family of distributed, in-memory, scalable solutions that enables you to build highly scalable and responsive applications by providing you super-fast access to your data. We offer the following types of Azure Cache:

  • Azure Redis Cache: Built on the open source Redis cache. This is a dedicated service, currently in Preview.
  • Managed Cache Service: Built on App Fabric Cache. This is a dedicated service, currently in General Availability.
  • In-Role Cache: Built on App Fabric Cache. This is a self-hosted cache, available via the Azure SDK.

As you can see from the description above, first and foremost the decision is between two technologies – Redis and AppFabric caching. Then, as things stand currently, for AppFabric there’s another choice of deployment model  – managed or in-role. (I suspect that one can create an in-role Redis cache with not too much effort, but this is not currently an out-of-the-box feature and so I’ll ignore it).

My experience with AppFabric Caching is a good one – I find it performs well, feature rich and mature for the enterprise. I have not had the chance to work with Redis but the feedback that I am getting from colleagues and customers is also very good, it seems it performs slightly better than AppFabric in many (but not all) scenarios, and it has some interesting features beyond what AppFabric provides.

I won’t go into a detailed comparison between the two (although unfortunately I can’t find a recent one either) but that feedback leads me to think that the first point to consider is your application requirements from its caching platform and whether that leads towards one solution over another.

All things being equal, at this point in time, I find that AppFabric provides a bit more flexibility compared to Redis in the sense that I can use it as a managed service on Azure, I can create a private cache in the cloud using in-role caching and I can run it locally in my data centre with a reasonably straight forwarded deployed.

If this flexibility is important to you than, right now, AppFabric might be the better choice.

And finally – if you have settled on using AppFabric on Azure – should you use the managed cache service or in-role service?

My starting position is always to prefer the as-a-service option. I have little if anything to manage, I don’t need to worry about the impact of updates to my roles or to balance resources between my roles and the cache.

So – when do customers prefer in-role caching? in my experience two arguments come up – 1: when they are not happy with a cache service accessible via the internet. using In-role caching I have more control over who has access to my cluster.  2: when they want to save the cost and believe they have enough capacity to balance role memory and in-role caching

Caution required with BizTalk’s SB-Messaging adapter

BizTalk 2013 ships with a native adapter for the Windows Azure Service Bus – the SB-Messaging adapter.

The adapter provides a very easy to use approach to receiving and sending messages to and from a service bus queue/topic/subscription but, as it turns out, it might be that it is a little bit simplistic or – at the very least – some caution is required as I’ve learnt with a customer yesterday –

The customer uses BizTalk to read a message from a Service Bus queue using the adapter. In the pipeline they ‘debatch’ the incoming message into multiple messages that get published to BizTalk’s message box.

This all worked just fine during testing, but when the customer moved to performance testing they hit a point after which they got a lot of errors such as the follows –

The adapter "SB-Messaging" raised an error message. Details "System.ObjectDisposedException: Cannot access a disposed object.

Object name: ‘Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement+RedirectContainerChannelFactory`1[System.ServiceModel.Channels.IRequestSessionChannel]‘.

The adapter "SB-Messaging" raised an error message. Details "Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.

When reading from the Service Bus the client can employ one of two strategies – a desructive read whereby the message is remove from there queue as soon as it is being read or a ‘peek lock’ strategy where the read message is locked and effectively hidden on the queue for a duration, allowing the client to process the message and then come back and either confirm that it had successfully processed the message or abandon the message, effectively  putting it back on the queue.

The peek-lock strategy is very useful to ensure no message loss and BizTalk’s SB-Messaging adapter makes a good use of it – until the received message is persisted in the message box it is not removed from the queue and hence zero-message loss can be guaranteed.

On the flip side, it means that a message may be delivered more than once and this has to be catered for.

In this customer’s case, the debatching they performed meant that, under load, it took BIzTalk too much time to complete the debatching and persisting all the messages to the message box. That meant that it did not go back to the queue in time to release the lock resulting with the error above. unfortunately this also caused a snow-ball effect as it meant that another BizTalk instance picked up the same message and the issue repeated itself, with the large number of messages being persisted to the message box increasing the load on the server and with it the duration of the receive pipeline processing….

Once identified they have found a quick workaround, at least in the first instance – remove the debatching from the service bus receive port and move it to a later stage in their BizTalk processing. This, I think, is a good appraoch, as it lets them ‘take ownership’ of the message in BizTalk, removing it from the Service Bus, before starting the heavy processing.

Another approach is to change the lock duration property on the queue/subscription in question. the default is 60 seconds. This could by time and with careful load-testing a reasonable value may be found, but – of course – it does not eliminate the risk, just pushes it out further.

Asking in the virtual corridors of Solidsoft it appears we have seen this issue with at least  two other customers. I think that the SB-Messaging adapter should support both strategies and allow the customer the choice, and perhaps flagging the need to set the lock duration value on the queue/subscription as required.

 

Cross posted on the Solidsoft blog

Caution required with BizTalk’s SB-Messaging adapter

BizTalk 2013 ships with a native adapter for the Windows Azure Service Bus – the SB-Messaging adapter.

The adapter provides a very easy to use approach to receiving and sending messages to and from a service bus queue/topic/subscription but, as it turns out, it might be that it is a little bit simplistic or – at the very least – some caution is required as I’ve learnt with a customer yesterday –

The customer uses BizTalk to read a message from a Service Bus queue using the adapter. In the pipeline they ‘debatch’ the incoming message into multiple messages that get published to BizTalk’s message box.

This all worked just fine during testing, but when the customer moved to performance testing they hit a point after which they got a lot of errors such as the follows –

The adapter "SB-Messaging" raised an error message. Details "System.ObjectDisposedException: Cannot access a disposed object.

Object name: ‘Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement+RedirectContainerChannelFactory`1[System.ServiceModel.Channels.IRequestSessionChannel]‘.

The adapter "SB-Messaging" raised an error message. Details "Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.

When reading from the Service Bus the client can employ one of two strategies – a desructive read whereby the message is remove from there queue as soon as it is being read or a ‘peek lock’ strategy where the read message is locked and effectively hidden on the queue for a duration, allowing the client to process the message and then come back and either confirm that it had successfully processed the message or abandon the message, effectively  putting it back on the queue.

The peek-lock strategy is very useful to ensure no message loss and BizTalk’s SB-Messaging adapter makes a good use of it – until the received message is persisted in the message box it is not removed from the queue and hence zero-message loss can be guaranteed.

On the flip side, it means that a message may be delivered more than once and this has to be catered for.

In this customer’s case, the debatching they performed meant that, under load, it took BIzTalk too much time to complete the debatching and persisting all the messages to the message box. That meant that it did not go back to the queue in time to release the lock resulting with the error above. unfortunately this also caused a snow-ball effect as it meant that another BizTalk instance picked up the same message and the issue repeated itself, with the large number of messages being persisted to the message box increasing the load on the server and with it the duration of the receive pipeline processing….

Once identified they have found a quick workaround, at least in the first instance – remove the debatching from the service bus receive port and move it to a later stage in their BizTalk processing. This, I think, is a good appraoch, as it lets them ‘take ownership’ of the message in BizTalk, removing it from the Service Bus, before starting the heavy processing.

Another approach is to change the lock duration property on the queue/subscription in question. the default is 60 seconds. This could by time and with careful load-testing a reasonable value may be found, but – of course – it does not eliminate the risk, just pushes it out further.

Asking in the virtual corridors of Solidsoft it appears we have seen this issue with at least  two other customers. I think that the SB-Messaging adapter should support both strategies and allow the customer the choice, and perhaps flagging the need to set the lock duration value on the queue/subscription as required.

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: