Sample of custom PowerShell cmdlets to manage Azure ServiceBus queues

A customer I’m working with these days is investing in automating the provisioning and de-provisioning a fairly complex solution on Azure using PowerShell and Azure Automation to help lower their running costs and provide increased agility and resiliency.

Within their solution they use Azure Service Bus Queues and Topics and unfortunately there are no PowerShell cmdlets to create and delete these.

Thankfully, creating PowerShell cmdlets is very easy and with the rich SDK for Service Bus creating a few cmdlets to perform the key actions around queue provisioning and de-provisioning took very little time.

In this post I wanted to outline the process I went through and share the sample I ended up producing.

First, I needed to get my environment ready – – to develop a custom PowerShell cmdlet I needed a reference to System.Management.Automation.

This assembly gets installed as part of the Windows SDK  into the folder C:\Program Files (x86)\Reference Assemblies\Microsoft\WindowsPowerShell  so I installed the SDK for Windows 8.1 which can be found here.

Once installed, I created a new Class Library project in Visual Studio and added the reference.

Given that I’ve never developed a cmdlet before I started with a very simple cmdlet to wrap my head around it, turns out that creating a simple cmdlet that returns the list of queues in an Azure Service Bus namespace is incredibly simple –

 [Cmdlet(VerbsCommon.Get, "AzureServiceBusQueue")]
    public class GetAzureServiceBusQueue : Cmdlet
    {
        protected override void EndProcessing()
        {
            string connectionString = string.Format("Endpoint=sb://{0}.servicebus.windows.net/;SharedAccessKeyName={1};SharedAccessKey={2}", Namespace, SharedAccessKeyName, SharedAccessKey);
            NamespaceManager nm = NamespaceManager.CreateFromConnectionString(connectionString);
            WriteObject(nm.GetQueues(), true);
        }
    }

I created a class and inherited from CmdLet in the System.Management.Automation namespace and added the Cmdlet attribute. it is the latter that provides the actual Cmdlet name in the verb-noun structure.

 

I then added an override for EndProcessing which creates an instance of the Service Bus’ NamespaceManager class with a connection string (in the code above I obviously removed the members I added to hold the hardcoded credentials, you will need to add these), used it to retrieve all the queues in the namespace and wrote them as the output from the CmdLet using the base class’ WriteObject method.

The true parameter tells PowerShell to enumerate over the responses.

Once compiled I used Import-Module to load my cmdlet and call it using Get-AzureServiceBusQueue. The result was an output of all the properties of the queues returned. magic.

Ok – so the next step was obviously to stop hardcoding the connecting string details – I need a few properties for my CmdLet –

I removed my hard-coded members and added properties to the class as follows –

        [Parameter(Mandatory = true, Position = 0)]
        public string Namespace { get; set; }

        [Parameter(Mandatory = true, Position = 1)]
        public string SharedAccessKeyName { get; set; }

        [Parameter(Mandatory=true, Position=2)]
        public string ShareAccessKey { get; set; }

Mandatory is self-explanatory. Position allows the user to pass the parameters without specifying their name but by specifying them in the correct order. I could now use this CmdLet in two ways

Get-AzureServiceBusQueue –Namespace <namespace> –SharedAccessKeyName <key name> –SharedAccessKey <key>

or

Get-AzureServiceBusQueue <namespace> <key name> <key>

both yield exactly the same result.

Next I knew I needed to add more CmdLets – to create and remove queues – for example, and it seemed silly to re-create a namespace manager every time.

The next logical step was to create a CmdLet that created a Namespace Manager and then pass that into any other queue-related CmdLet, I started by creating the Get-AzureServiceBusNamespaceManager CmdLet as follows –

[Cmdlet(VerbsCommon.Get,"AzureServiceBusNamespaceManager")]
    public class GetAzureServiceBusNamespaceManager : Cmdlet
    {
        [Parameter(Mandatory = true, Position = 0)]
        public string Namespace { get; set; }
        [Parameter(Mandatory = true, Position = 1)]
        public string SharedAccessKeyName { get; set; }
        [Parameter(Mandatory=true, Position=2)]
        public string ShareAccessKey { get; set; }
        protected override void EndProcessing()
        {
            base.EndProcessing();
            string connectionString = string.Format("Endpoint=sb://{0}.servicebus.windows.net/;SharedAccessKeyName={1};SharedAccessKey={2}",Namespace,SharedAccessKeyName,ShareAccessKey);
            WriteObject(NamespaceManager.CreateFromConnectionString(connectionString));
        }
    
    }

With this in place I removed the creation of  the NamespaceManager from the Get-AzureServiceBusQueue CmdLet and added a parameter, it now looked like this –

 [Cmdlet(VerbsCommon.Get, "AzureServiceBusQueue")]
    public class GetAzureServiceBusQueue : Cmdlet
    {
        [Parameter(Mandatory = true, ValueFromPipeline = true)]
        public NamespaceManager NamespaceManager { get; set; }
        protected override void EndProcessing()
        {
            WriteObject(NamespaceManager.GetQueues(), true);
        }
    }

I made sure to set the ValueFromPipeline property of the Parameter attribute to true; this allows me to pass the namespace manager as an object down the powershell pipeline and not just as a parameter. It meant I had two ways to use this CmdLet

Get-AzureServiceBusNamespaceManager <namespace> <sas key name> <sas key> | `
Get-AzureServiceBusQueue

or using two separate commands –

$namespaceManager - Get-AzureServiceBusNamespaceManager <namespace> <sas key name> <sas key> 
Get-AzureServiceBusQueue -Namespace $namespaceManager

The last bit I wanted to add is an optional parameter allowing me to specify the path of the queue I’m interested in so that I don’t have to retrieve all the queues all the time.  I also moved the code from EndProcessing to ProcessRecord which means it will get called for any item piped in. in theory that allows for a list of namespace managers to be pushed in, although i don’t really see that’s happening. The final CmdLet looks like this –

[Cmdlet(VerbsCommon.Get, "AzureServiceBusQueue")]
    public class GetAzureServiceBusQueue : Cmdlet
    {
        [Parameter(Mandatory = false, Position = 0)]
        public string[] Path { get; set; }

        [Parameter(Mandatory = true, ValueFromPipeline = true)]
        public NamespaceManager NamespaceManager { get; set; }
        protected override void ProcessRecord()
        {
            base.ProcessRecord();

            IEnumerable<QueueDescription> queues = NamespaceManager.GetQueues();
            if (null != Path)
                queues = queues.Where(q => Path.Contains(q.Path));

            WriteObject(queues, true);
        }
    }

Actually, as I started to think about the next few Cmdlets I’ve realised they would all need the namespace manager parameter, so I extracted that to a base class and inherited from that.

With this done, I added further Cmdlets to remove and create queues (for the latter, I’ve added a few more properties) –

[Cmdlet(VerbsCommon.Remove, "AzureServiceBusQueue")]
    public class RemoveAzureServiceBusQueue : AzureServiceBusBaseCmdlet
    {
        [Parameter(Mandatory = true, Position = 0)]
        public string[] Path { get; set; }

        protected override void ProcessRecord()
        {
            base.ProcessRecord();
            foreach (string p in Path)
            {
                if (NamespaceManager.QueueExists(p))
                    NamespaceManager.DeleteQueue(p);
                else
                    WriteError(new ErrorRecord(new ArgumentException("Queue " + Path + " not found in namespace " + NamespaceManager.Address.ToString()), "1", ErrorCategory.InvalidArgument, NamespaceManager));
            }
        }
    }
 [Cmdlet("Create","AzureServiceBusQueue")]
    public class CreateAzureServiceBusQueue : AzureServiceBusBaseCmdlet
    {
        [Parameter(Mandatory=true,Position=0)]
        public string[] Path { get; set; }
        [Parameter(HelpMessage="The maximum size in increments of 1024MB for the queue, maximum of 5120 MB")]
        public long? MaxSize { get; set; }

        [Parameter(HelpMessage="The default time-to-live to apply to all messages, in seconds")]
        public long? DefaultMessageTimeToLive { get; set; }

        bool enablePartitioning;
        [Parameter()]
        public SwitchParameter EnablePartitioning
        {
            get { return enablePartitioning; }
            set { enablePartitioning = value; }
        }
        protected override void ProcessRecord()
        {
            base.ProcessRecord();
            foreach (string p in Path)
            {
                QueueDescription queue = new QueueDescription(p);

                if (MaxSize.HasValue)
                    queue.MaxSizeInMegabytes = MaxSize.Value;
                if (DefaultMessageTimeToLive.HasValue)
                    queue.DefaultMessageTimeToLive = new TimeSpan(0, 0, 0, int.Parse(DefaultMessageTimeToLive.Value.ToString()));
                queue.EnablePartitioning = enablePartitioning;
                NamespaceManager.CreateQueue(queue);
            }
        }
    }

Of course, this is by no means complete, but it does perform all 3 basic operations on queues and can easily be expanded to support everything that the customer requires to support their automation needs

Advertisements

Azure CmdLets error: …no endpoint listening…

This had now happened too many times and I can never remember, so here’s a mental note –

When using PowerShell to create a VM on Azure (I suspect this would apply in other circumstances too), I occasionally bump into this error

New-AzureVM : There was no endpoint listening at https://management.core.windows.net/<subscription id>/services/storageservices/<storage account> that could accept the message. This is often caused by an incorrect address or SOAP action.

The important pieces of information in this message is the first GUID I’ve replaced here with <subscription id> and the second which I’ve replaced with <storage account> – this determines which storage endpoint will be used when creating the VM and are picked up according to the parameters used for the Set-AzureSubscription I’ve blogged about previously

The error above is displayed typically when one of these parameters are not set or set incorrectly. if you haven’t used Set-AzureSubscription the cmdlet will not know where to look for the endpoint.

Equally, I’ve managed in one or two occasions to mistakenly set an inompatible subscription id and storage account , that is – set subscription id ‘A’ with a storage account that belongs to subscription ‘B’ – this also, of course, cannot be resolved and will raise the same error.

Backing up a Windows Azure Virtual Machine using PowerShell

Back in June I wrote about how to use snapshots to backup and restore VHDs used by Windows Azure Virtual Machines.

Whilst this approach does have the limitation of only copying the state persisted to the disk and so I would not recommend using it on running VMs, for cases when the VM can be shut-down temporarily this can be a handy approach.

This could be particularly useful for dev and test scenarios when the VMs can be turned off and when users of those VMs might want to be able to go back to various known states.

Of course doing the whole process manually is cumbersome, time consuming and error prone, so scripting is needed and so in this post I’ll show what it takes to do this using PowerShell

‘Backing up’ a virtual machine using snapshots

The first job would be to turn off the virtual machine we want to back-up

Stop-AzureVM -ServiceName $cloudSvcName -Name $vmname

With the VM stopped it is time to take the snapshot. unfortunately the Azure PowerShell Cmdlets do not, currently, support snapshot operations on Blobs but thankfully Cerebrata have produced a great set of Cmdlets for Azure Mananagement

Taking a snapshot of a Blob using the Cerebra Cmdlets is very easy –

Checkpoint-Blob -BlobUrl $blobUrl -AccountName $accountName -AccountKey $accountKey

It is worth noting that this command returns the url for the snapshot, and it might be useful to keep for later use. Also worth noting is that it is possible to add metadata to the snapshot, in the form of a collection of key-value pairs, which could be used to select snapshots later, finally it is useful to know that this command is pretty much instantaneous  so whilst there’s some downtime required for shutting down and starting up the machine, taking the snapshot takes no time at all.

With the snapshot taken it is now time to start our VM again

Start-AzureVM -ServiceName $cloudSvcName -Name $vmname

and so, with a very short powershell script, one can create an effective backup of a virtual machine which will be stored with the Blob. It is worth noting again that this is only reliable when the machine is stopped (turned off) before the snapshot is taken. it is also worth noting that if machines may have more than one disk and additional disks may need to be handled too (in the same way)

‘Restoring’ a virtual machine using snapshots

Restoring a VM is a little bit more involving – when a disk is attached to a VM a lease is created on the Blob storing the disk effectively locking it for everyone else.

That means that one has to remove the VM before a snapshot can be ‘promoted’ on a VHD (effectively overwriting the blob with the snapshot), stopping the VM would not be enough, and so to start we do have to stop the VM again, as before –

Stop-AzureVM -ServiceName $cloudSvcName -Name $vmname

but in order to update the OSDisk we also need to remove it completely.

Before we do that, given that we’re going to bring it back online shortly, it would be useful to export all of its configuration first –

Export-AzureVM -ServiceName $cloudSvcName  -Name $vmname  -Path 'c:\tmp\export.xml'

and with the VM configuration exported, we can go ahead and remove it

Remove-AzureVM -ServiceName $cloudSvcName -Name $vmname

Now that the machine is gone, its lease on the blob is gone too, which means we should be able to use Cerebrata’s Copy-Blob Cmdlet to copy the snapshot over the Blob

Copy-Blob -BlobUrl $blobUrl -TargetBlobContainerName $container -AccountName $accountName -AccountKey $accountKey

Like the Checkpoint-Blob command this is pretty much instantaneous.

Now that we have the updated disk in place, it’s time to bring our VM back to life, first import –

Import-AzureVM -Path 'c:\tmp\export.xml' | New-AzureVM -ServiceName $cloudSvcName -Location 'North Europe'

and that is it – when the VM creation, using the imported configuration, completes – we have our VM back online this time running off the updated VHD state

Final note – in this post I just wanted to show the key operations one would use to create a useful script. the actual process will vary depending on the specific requirements and , of course, it is possible  to make quite an elaborate script with PowerShell – for example it is possible to enumerate on all the VMs in a particular cloud service and for each VM query the OS and Data disks and then perform all the operations above over these enumerations, the sky is pretty much the limit! 🙂

Azure Subscriptions in PowerShell demystified

Delivering an Azure workshop at Microsoft yesterday, where we’ve been doing lots of IaaS and PowerShell work, a good question came up –

When you run Get-AzureSubscription – where does it get the subscription information from?

I guess the underlying question is really – how do the Windows Azure PowerShell Cmdlets relate to accounts, subscriptions and the like?

Well – when you first start the Windows Azure PowerShell console, if, for example, you try to get the list of subscriptions available to you using Get-AzureSubscription you get the following error

image

This is the CmdLet telling you it does not have any publishing settings to work with.

Just like Visual Studio, any client wishing to work with the Windows Azure Management API needs to have credentials set; generally speaking this involves creating and uploading a certificate to the management portal and then configuring the client to use this certificate to authenticate.

Thankfully Microsoft had kindly provided a way to automate this process somewhat using publishsettings files. one can browse to http://windows.azure.com/download/publishprofile.aspx (from PowerShell, using Get-AzurePublishSettingsFile will open the brower with that url for you :-)), sign in using the credentials associated with one or more Azure subscriptions and the page will –

a) Create a self signed certificate
b) Configure the certificate as a management certificate for each subscription the signed-in account is an admin of
c) Download a publishsettings file with the details of the certificate and all the existing subscriptions for the account

Here’s a screen shot showing the certificate configured against a couple of subscriptions I have –

image

The downloaded publishsettings file can then be imported in Visual Studio or, indeed, in PowerShell using the Import-AzurePublishSettingsFile CmdLet

So – what does this CmdLet do?

Well – as mentioned above – the publishsettings file includes a couple of things – the management certificate used and a list of the subscriptions available for the account  – and so the CmdLet starts by importing the certificate into the User’s personal certificate store.

It then goes on to create a couple of files in the user PowerShell profile data file (located in %appdata%\Windows Azure Powershell) –  publishsettings.xml which is very similar to the downloaded publishsettings file, only that instead of holding the entire certificate it now only holds the thumbprint (as the certificate is securely stored in the certificate store)

Note: if it wasn’t clear before, it should now be, that the downloaded file should be properly deleted as soon as possible or stored in a secure place as anyone who gains access to it gains administrative access to the related subscriptions!

The second file – DefaultSubscriptionData.xml – also lists the available subscriptions and the associated certificates’ thumbprints, more on that file later.

You can see from the above that repeatedly downloading and importing the publishsettings file is not a good habit as it will keep generating and storing certificates and indeed the portal will only allow 10 certificates per subscription and will error after that, so this process should really only be done once per user and existing certificates, transferred out of band, are a better way to set things up for larger teams.

To that extent Set-AzureSubscription can be used to provide details of existing subscriptions without using a publishsettings file providing that you have access to the management certificate, so if the management certificate had been distributed out-of-band to the client machine Set-AzureSubscription can be used to add it to the user’s context without using Import-AzurePublishSettingsFile and without generating and configuring new certificates in the portal

Once set, the details are kept in the DefaultSubscriptionData.xml file, so future PowerShell sessions will re-use the subscriptions set, using either method.

One subscription is marked as the default subscription, and will be used by default for all Azure related commands unless Select-AzureSubscription is used to select another one; Set-AzureSubscription can also be used to set a different default.

In addition – Remove-AzureSubscription can be used to remove a subscription’s details from the context, this will remove them from the DefaultSubscriptionData.xml file.

Last, but not least, one can use the –SubscriptionDataFile switch on these commands to work against a file other than the DefaultSubscriptionData.xml file which allows one to manage even a large set of subscriptions conveniently; you can split this however you like, I can see people doing this by project, by environment (dev/test/prod), but customer (for consultants) etc.

%d bloggers like this: