Extending SharePoint Online with Windows Azure

As customers move to O365 and SharePoint Online (SPO) and begin to realise the benefits of public cloud, they typically look to expand the scope of their cloud deployment further and often extensions to SharePoint is the most obvious place to look. after all – if,  you’ve taken the decision to move your SharePoint deployment to the cloud, why wouldn’t you want to host extensions to it there too? Windows Azure simply makes perfect sense in this case.

There are various ways in which Azure can be used to enhance SPO, amongst them –

• One can serve data through Windows Azure (from SQL Database, for example) which can be displayed in SPO as an ‘external lists’
• One can host a process on Azure, typically as a Worker Role, that would read and/or write from SharePoint lists alongside some other processing
• One can surface a custom application, deployed on Windows Azure, through the SPO UI

In this post I want to demonstrate how simple it is to do the latter, and to make it useful ensure that I’m adding Single-Sign-On to the mix.

To start with, I deployed a vanilla ASP.net web role project onto my subscription.

Browsing to the newly deployed cloud service looks like this –

On SharePoint online, I went to our team’s site and added a new page so that I can experiment with without getting on anyone’s nerves

On the page that was created I added a ‘Page Viewer Web Part’ and in the part’s properties I set the URL to my cloud service

The result was the Azure content displayed within the portal

Note: one side-effect is a warning by IE (“Only secure content is displayed”) which has to be acknowledged in order for the Azure content to be displayed. This is due to the fact that SharePoint Online is served over SSL and my web role isn’t. adding a valid SSL endpoint to my web role avoids this issue (but may cause another if you’re indeed using a self-signed certificate )

So – considering that my Windows Azure deployed application could have been something actually meaningful and useful, it’s handy to know that serving that through the corporate portal on O365 is a nice-and-easy (circa 10 minutes!), but of course, as I mentioned, to be a realistic scenario I also had to make sure I plumb it all to our identity scenario ensuring that my Azure-based application can be secured without hassling the users to authenticate again.

In our case SPO uses ADFS against our on-premises AD, and that model, not-surprisingly, maps very well to Windows Azure.

In order to integrate my Windows Azure based application with ADFS I started by creating an Access Control Service Namespace and in that I had added our ADFS as an identity provider. this takes a few minutes to set-up requires access to the ADFS metadata (URL or file) as well as setting up ACS as a relying party in ADFS. it is also important to add at least one claim rule to ensure ADFS actually emits some claims to ACS and through that to your application.

With the ACS->ADFS link done it was time to add my application as a relying party in ACS and generate/configure the rules for the claims, in my case I was happy to pass the claims from ADFS as is, but I also, for testing purposes, add rules to populate the name output claim with the windowsaccountname claim from ADFS.

With all the configuration done it was time to integrate my application with ACS, which is easily done using the Identity and Access tool in Visual Studio,  selecting my ACS namespace and identity provider I configured earlier and I was pretty much ready to test my setup.

To be able to verify the sign-in worked I added a couple of controls to my page that will confirm the user is logged in and will display the user-name.

   <asp:loginstatus ID="Loginstatus1" runat="server"></asp:loginstatus>
    <asp:LoginName ID="LoginName1" runat="server"/>

I went ahead and tested it straight on Windows Azure. if you choose to test it locally first (which saves the deployment time) be sure to update the configuration with the correct (local) Application URL, and then change it again when deploying to Azure.

Sure enough, with the federated identity configuration in place, browsing to my application from within our corporate network provides me with a seamless single-sign-on experience and my login-details appear on the frame within the SharePoint page.

Outside the network I can still get to SharePoint Online and to my application, but I am required to enter my credentials the first time around, which will then be cached according to the settings in my application configuration file and ACS.

All of this means not only that I have secured access to my Windows Azure based application by requiring authentication and that I made it easy for my colleagues by employing Single-Sign-On with our corporate identity, I can, of course, now use standard .net mechanism to manage authorisation across the application 

BizTalk Enterprise on Azure

With Azure IaaS and BizTalk 2013 officially released I wanted to do a quick exercise of setting up a multi-server BizTalk environment.

I started by creating a virtual network and deployign a domain controller onto it, I then added a SQL Server instance from the platform images and joined it to the domain and finally I added a BizTalk Enterprise instance from the platform images and joined that to the domain too.

So far everything went smooth. I then nearly remembered that in a multi server environment I needed to manually create the necessary domain groups and that was easy enough to do but still – I could not get the BizTalk configuration Wizard to work.

Ok – at least partially this was due to my own arrogance thinking I can still remember how to do this by heart despite having not touched BizTalk in anger for over two years, but partially it is simply the Conifugraion Wizards unhelpful approach to error reporting.

Either way thrawling through the log file revealed the key to finding out what’s missing – the error – “Failed to read "KeepDbDebugKey" from the registry”

A quick Bing search pointed out the infamous mistake – missing the step to enable DTC appropriately.

Quite handily the BTS image comes with DTC correctly configured (no surprise there) but as I was using a vanilla SQL Server image it had DTC disabled, and that needed changing

With that done I could properly configure BizTalk 2013 on an Azure IaaS instance and get everything to work

Cost efficiency through transparency

There are many aspects to the cost saving argument for cloud computing. in the past it was suggested too often that cloud could bring head-count related cost efficiencies. I never subscribed to that view and I no longer hear it mentioned as frequently. Cloud does bring a shift in the focus of existing resources, allowing them to invest more in where it matters – the application – and less in mundane tasks such as maintaining hardware, which is very handy indeed, in particular given that, in today’s economic climate, most organisation are having to run quite lean anyway, but that’s a topic for another post.

Other aspects of cost-saving to cloud computing can be more concrete and accurate, such the reduced cost of hardware and resources, the efficiencies in the very modern data centres, etc.

But there’s one aspect of cost saving Windows Azure brings, that, although not the biggest, is the one I find most interesting and one that can have a big impact on how software is conceived and designed, – the accuracy and transparency about the cost of running a specific solution or even individual solution components on the platform.

This is not wholly novel, but in most organisations I’ve had a chance to work with over the years, the cost of running IT was never broken down to a low enough level to allow management to estimate the cost of running any particular solution. In fact – in many organisations IT is budgeted annually and then, to some degree, that aspect is forgotten about, until the discussion of next year’s budget that is.

This comes up often in the early discussion about Windows Azure when the pricing calculator is used to estimate the cost of running any application on Azure, which is then contrasted with whatever estimates exist for the cost of running said application on-premises (or elsewhere).

Having a reasonably accurate view of the cost of a solution often forces a debate that otherwise would not have occurred or may only occur much later. This can be daunting at first, but over time this undoubtedly leads to more informed, and thus better decisions on what to run, how and when.

Similarly, when purchased through an Enterprise Agreement Windows Azure allows customers to set up their own subscriptions at a very granular level, used well, this can provide a very accurate, and almost in real time, view of the costs associated with each application, allowing the organisation to make better decisions on where to invest and at what scale and may even encourage businesses to try things they may have not tried otherwise.

For development teams a new, and quite interesting, aspect when designing solutions is introduced – designing for cost. one can get quite hooked up by this, which is a risk project managers will do well to manage closely, but with the clarity of the costs associated with solution deployed on PaaS platforms such as Windows Azure, one can consider multiple design options and optimise for costs as necessary. a consideration that rarely made it to the early iterations of a project before.

For example – knowing that every poll on a service bus queue to check if a message exists counts towards the cost, whether a message was found or not, coupled with a clear cost model for said queue, and with a platform that scale as necessary and automatically, one is equipped to decide whether implementing a truncated exponential backoff pattern is justified by the expected savings this would bring.

Of course, the total cost of ownership of an application is bigger than just the development, hardware and licenses, and factoring elements such day to day operations and maintenance is harder; more so with seemingly less known paradigms such as cloud computing, where operations could, and should, be done slightly differently, but the former are a good starting point and models begin to emerge to help organisation define the cost cost of running cloud solutions based on their size and componentry and these would help as will experience, of course.

Managers also have to consider the new balance between the options available to them – facing with a throughput problems, for example, managers need to consider whether tasking a developer with improving the performance of an application is more cost-effective than scaling the application out further. the latter option rarely stacked up before public cloud, but now makes perfect sense.

The transparency that Windows Azure brings into the cost of running a solution can have an effect on several aspects of software development, each may lead, on its own right, to cost savings.

Grown up cloud or – why I moved to Solidsoft

With the adoption and maturity of public cloud growing, businesses are changing the critical lenses they use when thinking about deploying applications to the cloud.

During the last couple of years, during which I delivered countless briefings on Windows Azure to businesses across the UK, I found that, broadly speaking, they typically fall into one of three buckets – those who are jumping in head first (a clear minority, mostly composed of start-ups and very small ventures), those who object completely (another minority who see public cloud as too new, to risky, to flaky, too different, not different enough, or any number of other reasons) and the vast majority who wanted to explore further, albeit with some caution.

This third group, the majority, and a very diverse group, wanted to criticise public cloud from a point of knowledge, to get some first-hand experience, and it’s members usually sought a project or two which were small enough and contained enough, which provided a fairly narrow and well defined profile of risk, and gave it a go. some took it further than others, but, generally speaking, failure was almost always kept as an option and the cloud was treated, to a degree, at an arms length.

I’m a curious, yet conservative, guy, and I can subscribe to this view very well – a child who’s been given a new toy and wants to poke it around a bit, see what it does, see if it’s fun, before deciding to its worth welcoming this new toy to the group of favourites, maybe even chucking away some of the old ones to make some room. It all makes perfect sense.

This poking around business, though, meant that, until recently, most have taken a fairly uncommitted approach to public cloud and have not yet started to look at what it means to take public cloud seriously. They have not yet made place in the pram for the new toy.

I sense that there has been a real change in this respect recently, driven partly by the maturity of the public cloud platforms and the surrounding technologies but largely because businesses, who have seen the benefits of the cloud either first hand or by observing others, are now starting to seriously commit to adopting public cloud as mainstream.

What this means is that the scope of the discussion around cloud computing has to be much broader than it has been so far, and the proposed solutions have to be more comprehensive.

These ‘grown up’ cloud solutions need to ensure that, with the great benefits that public cloud bring in cost, flexibility, agility, collaboration, focus, etc, the other aspects of the enterprise IT landscape such manageability, supportability, end-to-end security, etc all looked into in detail as well.

Grown up cloud takes into deep consideration the fact that what gets deployed into the cloud has to remain deeply connected with everything else – whether it is integrating parts of a solution that spans cloud and on-premises, integrating an application in the cloud to the rest of the enterprise or indeed connecting together multiple cloud tiple platforms into one coherent end-to-end solution.

Be it EAI and/or B2B, or a single application that spans cloud and on-premises, one has to think about how a versioning strategy would looks like, how would an IT estate that spans my data-centre and multiple clouds be managed? how does logging monitoring look like? what’s the back-up strategy? all these questions, and others, have to be looked at as part of a mainstream solution in the enterprise, cloud-or-not.

Of course this sounds very old-school, it is. but it’s, arguably, carries increased importance as organisation not only deploy applications to public cloud in an ever increasing velocity (and ferocity?) but actually as there’s a big move from build to buy, from manage to subscribe, all drive not only a move to a cloud platform but, indeed, to multiple cloud platforms.

In my personal story, this is exactly why I was delighted to accept a position as a principal consultant in Solidsoft’s CTO office, after a couple of great years in the Windows Azure space at Microsoft.

My role as a technical specialist for Windows Azure was absolutely brilliant, I love Microsoft as a company and as a place to work at, and I absolutely love the technology it delivers. I enjoyed (almost) every bit of my two years at Microsoft, but, for me, there was one big downside at this point in time – in this particular role, one does not get to walk along ones customers long enough to touch on everything that matters.

In that role, you work with customers as they make their first steps – you work with them to define their needs, you bring your understanding of the platform and mix it all together to get a view of a solution on the platform, at which point you, typically, pull out of the conversation somewhat and let the customer, often with the help of a technology partner such as Solidsoft, get on with delivering the vision.

Admittedly this has many advantages, but it also carries some disadvantages – and at this exciting time for software development in the enterprise I wanted to be firmly in this other part of the conversation – where both functional and non-functional requirements are digested in detail and translated into a complete solution, which is then built in a way that ensures it is is fully integrated with the rest of the enterprise fabric, that delivers long term benefits to the business, and that just works, and I could not think of a better place in the UK to do just that other than Solidsoft.

Hide-and-seek with Cloud Services containers for Virtual Machines on Azure

From several conversations I had recently it transpired that there’s some confusion around the relevancy of Cloud Service when using Virtual Machines on Azure.
Initially it surprised me, but then it was pointed out to me that, in fact, the Windows Azure Management Portal does a very good job festering this confusion.

I’ll get to why in a moment but first let me start with this -

The simplest way to think about how a Virtual Machine is deployed to Azure is the good-old VM Role -

Just as the VM Role used to, a Windows Azure Virtual Machine is deployed as a role with a single instance within a cloud service. This is quite apparent when you look at it in the old portal – here’s an example of a VM I created in the Management Portal using the standard quick-create as seen on the old portal -

Ok – there’s plenty behind the scenes that the platform is doing differently when managing VMs vs. Web or Worker roles, but from a deployment model point of view, a cloud service is created with a single instance of a role which happens to be a Virtual Machine.

So – why is the portal confusing? well – because, and I suspect that it was an attempt to simplify things that ended up making things rather more confusing – when you deploy a VM, the management portal chooses to hide it’s existence -

I’m guessing that the thinking was that with a single VM the cloud service is rather irrelevant – conceptually, the role and the instance are one and all the management operations and monitoring are done at an instance level and so - there was no real need to surface the fact that a cloud service exists.

Unfortunately, things start to get more confusing as soon as you delete your VM because at some point during the delete operation, the corresponding Cloud Service will suddenly appear, and eventually, when the delete operation will complete, it will be empty -

It appears largely for one reason – so that it can be deleted; otherwise a phantom service will remain in the subscription with no way to manage it through the UI.
BTW – the API and Powershell (and some dialogues, as you will see shortly), seem to always reveal the real details, and the cloud service that is clearly visible in the old-portal is also fully accessible through these. In fact – the description given to the cloud service says it all -

So – why the platform, which had kindly created this cloud service for me – doesn’t also kindly remove it? well – I suspect that this is because it cannot assume that the container will only contain the original VM. How come? let’s look at another case when the cloud service suddenly appears -

If you deploy a second VM, from Gallery, and choose to connect to an existing virtual machine -

(notice how the non-existent cloud service suddenly makes an appearance?)

What you are actually requesting the platform to do is to deploy an new instance, of a new role, into the same cloud service, and that’s a whole different story!  Now – not only the cloud service has a meaning that is greater than either VM definitions, deleting the first VM clearly cannot delete the cloud service and so – shortly after requesting the second VM - the cloud service will make it’s formal appearance in the portal -

and again – looking at how things look like in the old portal can help get a somewhat clearer behind-the-scenes view – one service, two roles, one instance of each -

In the new portal, this is reflected like this (which isn’t telling the full story) -

Interestingly, although not wholly surprisingly, removing the second VM does not re-hide the cloud service and so it is possible to ‘see in the wild’ a Cloud Service with a single VM in it -

And so – given this seemingly erratic behaviour, even if there is some method behind the madness, it is not surprising the people get confused.
Personally, I would have much preferred if things were straight forward, black and white – you create a VM, you get the corresponding Cloud Service. Simples.

But there you go – nobody asked me, so at least I hope this helps shed some light on the subject

Side note – all of this also helps explain the very long and repetitive names given to disks – the naming convention is [cloud serivce]-[VM name]-[disk number]-[timestamp]

Backing up a Windows Azure Virtual Machine using PowerShell

Back in June I wrote about how to use snapshots to backup and restore VHDs used by Windows Azure Virtual Machines.

Whilst this approach does have the limitation of only copying the state persisted to the disk and so I would not recommend using it on running VMs, for cases when the VM can be shut-down temporarily this can be a handy approach.

This could be particularly useful for dev and test scenarios when the VMs can be turned off and when users of those VMs might want to be able to go back to various known states.

Of course doing the whole process manually is cumbersome, time consuming and error prone, so scripting is needed and so in this post I’ll show what it takes to do this using PowerShell

‘Backing up’ a virtual machine using snapshots

The first job would be to turn off the virtual machine we want to back-up

Stop-AzureVM -ServiceName $cloudSvcName -Name $vmname

With the VM stopped it is time to take the snapshot. unfortunately the Azure PowerShell Cmdlets do not, currently, support snapshot operations on Blobs but thankfully Cerebrata have produced a great set of Cmdlets for Azure Mananagement

Taking a snapshot of a Blob using the Cerebra Cmdlets is very easy -

Checkpoint-Blob -BlobUrl $blobUrl -AccountName $accountName -AccountKey $accountKey

It is worth noting that this command returns the url for the snapshot, and it might be useful to keep for later use. Also worth noting is that it is possible to add metadata to the snapshot, in the form of a collection of key-value pairs, which could be used to select snapshots later, finally it is useful to know that this command is pretty much instantaneous  so whilst there’s some downtime required for shutting down and starting up the machine, taking the snapshot takes no time at all.

With the snapshot taken it is now time to start our VM again

Start-AzureVM -ServiceName $cloudSvcName -Name $vmname

and so, with a very short powershell script, one can create an effective backup of a virtual machine which will be stored with the Blob. It is worth noting again that this is only reliable when the machine is stopped (turned off) before the snapshot is taken. it is also worth noting that if machines may have more than one disk and additional disks may need to be handled too (in the same way)

‘Restoring’ a virtual machine using snapshots

Restoring a VM is a little bit more involving – when a disk is attached to a VM a lease is created on the Blob storing the disk effectively locking it for everyone else.

That means that one has to remove the VM before a snapshot can be ‘promoted’ on a VHD (effectively overwriting the blob with the snapshot), stopping the VM would not be enough, and so to start we do have to stop the VM again, as before -

Stop-AzureVM -ServiceName $cloudSvcName -Name $vmname

but in order to update the OSDisk we also need to remove it completely.

Before we do that, given that we’re going to bring it back online shortly, it would be useful to export all of its configuration first -

Export-AzureVM -ServiceName $cloudSvcName  -Name $vmname  -Path 'c:\tmp\export.xml'

and with the VM configuration exported, we can go ahead and remove it

Remove-AzureVM -ServiceName $cloudSvcName -Name $vmname

Now that the machine is gone, its lease on the blob is gone too, which means we should be able to use Cerebrata’s Copy-Blob Cmdlet to copy the snapshot over the Blob

Copy-Blob -BlobUrl $blobUrl -TargetBlobContainerName $container -AccountName $accountName -AccountKey $accountKey

Like the Checkpoint-Blob command this is pretty much instantaneous.

Now that we have the updated disk in place, it’s time to bring our VM back to life, first import -

Import-AzureVM -Path 'c:\tmp\export.xml' | New-AzureVM -ServiceName $cloudSvcName -Location 'North Europe'

and that is it – when the VM creation, using the imported configuration, completes – we have our VM back online this time running off the updated VHD state

Final note – in this post I just wanted to show the key operations one would use to create a useful script. the actual process will vary depending on the specific requirements and , of course, it is possible  to make quite an elaborate script with PowerShell – for example it is possible to enumerate on all the VMs in a particular cloud service and for each VM query the OS and Data disks and then perform all the operations above over these enumerations, the sky is pretty much the limit!

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

4,329 films were submitted to the 2012 Cannes Film Festival. This blog had 13,000 views in 2012. If each view were a film, this blog would power 3 Film Festivals

Click here to see the complete report.