Subscriptions

Before you can upload anything to Azure you need to make powershell aware of your Azure subscriptions.

http://yossidahan.wordpress.com/2012/12/12/azure-subscriptions-in-powershell-demystified/

These can be a minefield – especially if you have more than one and make a typo.

1) Delete any existing cached settings:  Delete the xml file from  ~\AppData\Roaming\Windows Azure Powershell

2) Get-AzureSubscription | select SubscriptionName, IsDefault (Should return nothing at this point)

3) Get-AzurePublishSettingsFile (asks you to logon and downloads a settings file with credentials)

4) Import-AzurePublishSettingsFile "C:\Temp\MSDNcredentials.publishsettings"

5) Get-AzureSubscription | select SubscriptionName, IsDefault (Should now return a list of your accounts and which is default)

Note: If you have spaces in any of your account names – its best to cut and paste from the output from Get-AzureSubscription to make sure its character perfect!

Prepare disk for upload.

If you have a local list that you have been using in Hyper V and you want to upload to Azure you need to first convert it (make sure there are no differencing disks etc for it).  Azure only supports fixed size (max 127GB) VHD disks (it doesn’t currently support vhdx disks)

If you need to convert a VHDX disk to a VHD you can do this using Convert-VHD

This HAS to be done using powershell on Windows 8 machine or later

Convert-VHD –Path c:\test\MY-VM.vhdx –DestinationPath c:\test\MY-NEW-VM.vhd

If you are just uploading a data VM then its relatively straight forward and the above should be sufficient.

If however you want to upload a bootable VM then there are a number of issues well described here: http://blogs.msdn.com/b/narahari/archive/2013/01/15/windows-azure-virtual-machines-gotcha-s.aspx

• The biggest issue about uploading VMs to Azure is that there is no Console connection in Azure, the only option for connections is via RDP. So if everything is exactly correctthere is a high chance that your new VM will be running but not accessible once you get it on Azure with no option to correct it!
• VHD has to be < 127 GB (Azure limit) – and thus applies to dynamic disk as well if the max size is > than that.It Maybe able to use VHDResizer to shrink the disk – but only if nothing is towards the end of the disk so it may need defrag first (see:http://morgansimonsen.wordpress.com/2009/03/27/reducing-the-size-of-a-dynamically-expanding-vhd-file/)
• The VM has to be generalised using sysprep – see: http://blogs.msdn.com/b/narahari/archive/2013/01/15/windows-azure-virtual-machines-gotcha-s.aspx.  A VM must be generalized to use as an image from which you will create other VMs

There is also an interesting looking tool here that is supposed to do the whole process – but on one quick try failed for me: http://cloudscraper.migrate2iaas.com/azuredownload

Uploading disk

Again using powershell.  The process is quite smart – although Azure only supports fixed sized disks -  it checks for only written blocks to minimise file transfer size:

• import-module azure
• select-azuresubscription "mysubscriptionname"   $sourceVHD = "D:\StorageDemos\myvhd.vhd"
• $destinationVHD = https://mwwestus1.blob.core.windows.net/uploads/myvhd.vhd 
• Add-AzureVhd -LocalFilePath $sourceVHD -Destination $destinationVHD ` -NumberOfUploaderThreads 5

You get a nice progress indicator in Powershell as it uploads

From Telehouse a 20GB disk (Dynamic Hyper V Server 2008 R2 vhd) took about an hour.

How to snapshot and restore Azure vhds in powershell

Good description here:

http://blog.cerebrata.com/how-to-snapshot-and-restore-virtual-machines-running-on-windows-azure/

Server 2012 and Server 2008 R2 are supported clients

Azure backup works using the Azure backup agent (downloadable from your Azure recover services, backup vault account.  Having installed this the first step is to link the server that its installed on to the Azure backup vault.  This requires a certificate.  There are options to self generate – but seeing as we had a proper one I thought I’d use that.

• Certificates must be imported into the local COMPUTER personal store.  This is different from you own user personal store
• To do this I had to be logged on as Administrator, then Start / Run MMC  and then Add Snap in for certificates and select personal computer
• Import pfx file here - making sure certificate ends up in Local computer personal store. Also make sure you import it with option to export

• Then export the certificate as a .cer file default option will do (Don’t think you need to export the key)

• In Azure – upload the certificate lo link this server to the azure backup account

• You can then run through the Wizard connecting Azure Backup agent to the backup store.  If all has gone well you will be offered your newly imported certificate to use.  If you have gone wrong – it will say no certificate available.

You will also be asked to generate a passphrase as the encryption key. Lose this and you will never recover your data if you need to re-install – so its suggested soring on USB stick securely.

Costs:

Currently first 5GB is free – then 0.18 pence per GB / month

Note – none of the details below are required for simply creating and using VMs in Azure directly.  The details come into play if you start ending to copy VHDs around and recreate VMs from them.

Install Azure Powershell

You need the command line Azure tools for PowerShell. Good description of getting started is here: http://azure.microsoft.com/en-us/documentation/articles/install-configure-powershell/

from here (download and install):

http://go.microsoft.com/fwlink/p/?linkid=320376&clcid=0x409

Will take a few minutes probably as it has a number of prerequisites that it may also have to fetch.  I also installed Windows Azure powershell

Eventually you will get a Windows Azure program folder and within that a Windows Azure Powershell shortcut

Connect to your subscription

I used the certificate method described here: http://azure.microsoft.com/en-us/documentation/articles/install-configure-powershell/

• Get-AzurePublishSettingsFile (opens browser here you log onto your Azure account and it generates and downloads a file for publishing settings)
• Import-AzurePublishSettingsFile  "C:\Temp\Microsoft Partner Network-Windows Azure  MSDN - Visual Studio Premium-6-11-2014-credentials.publishsettings"  (Import subscription settings)

Accessing your VMs

If all has gone well – Get-AzureVM should now list your VMs

There are options for starting and stopping – but I’d just use the Azure web pages normally.

Copying VHDs (Blobs) between storage accounts

From: http://michaelwasham.com/windows-azure-powershell-reference-guide/copying-vhds-blobs-between-storage-accounts/

Michael gives a nice overview of how Azure storage is structured and some indicative timings as to how long copies may take (for 127GB VHD 12 minutes within data centre but cross stamp and 23 minutes between data centres (- I suspect this depends on where the datacentres are)

This command will complete quickly – but the copy will take a while to complete – so see section below on checking progress

Select-AzureSubscription "Windows Azure  MSDN - Visual Studio Premium" 

### Source VHD (West US) - authenticated container ###

$srcUri = "https://[StorageName].blob.core.windows.net/vhds/Test.vhd" 

### Source Storage Account (West US) ###

$srcStorageAccount = [SourceAccountName]

$srcStorageKey = [SourceKey]

### Target Storage Account (West US) ###

$destStorageAccount = [DestinationAccountName]

$destStorageKey = [DestinationKey]

### Create the source storage account context ### 

$srcContext = New-AzureStorageContext  –StorageAccountName $srcStorageAccount `

                                        -StorageAccountKey $srcStorageKey  

### Create the destination storage account context ### 

$destContext = New-AzureStorageContext  –StorageAccountName $destStorageAccount `

                                        -StorageAccountKey $destStorageKey  

### Destination Container Name ### 

$containerName = "copiedvhds"

### Create the container on the destination ### 

New-AzureStorageContainer -Name $containerName -Context $destContext 

### Start the asynchronous copy - specify the source authentication with -SrcContext ### 

$blob1 = Start-AzureStorageBlobCopy -srcUri $srcUri `

                                    -SrcContext $srcContext `

                                    -DestContainer $containerName `

                                    -DestBlob "testD2.vhd" `

                                    -DestContext $destContext

I get an error – but the command seems to complete OK

Checking the progress of a blob copy

These can take a while – especially if between data centres – so its useful to be able to check how they are progressing.

$context = New-AzureStorageContext -StorageAccountName jamestestbackup –StorageAccountKey [your key]

Get-AzureStorageBlob -Container "copiedvhds" -Context $context | Get-AzureStorageBlobCopyState

Stopping an asynchronous blob

Copying blobs around can be tricky – especially if you are moving between accounts (and storage zones which makes things slow).

I managed to get some copies that seemed to appear in the destination – but were locked with a lease for a pending copy.  I managed to sop these pending copies by:

Select-AzureSubscription -SubscriptionName "Your Subscription Name"

$context = New-AzureStorageContext -StorageAccountName jamestestbackup -StorageAccountKey [your key]

Get-AzureStorageBlob -Container "copiedvhds" -Context $context| Stop-AzureStorageBlobCopy –Force

Creating a new VM from a BLOB

You can not do this until the copy is complete.  The Azure interface seems to give no indication of when this is (see above for powershell script).  If its not complete you will get an error about a lease being held (as the copy is still in progress) when you try and create the vhd

There are a couple of steps here:

1) You must use the VM, Disks section to create a VHD Disk from your BLOB (note your blob must have a .vhd file extension for you to be able to see it

2) Then you can create a new VM.  Use the “From Gallery” option and you will see an option bottom left that allows you to pick your own disks

Free tools

This can be useful as an alternative interface for working with blobs

http://www.cerebrata.com/download/azure-explorer

This allows you to connect to your Azure storage and copy BLOBS (vhds) around and download locally if you want.

IMPORTANT: If you copy from one Azure storage to another – the copy comes via your desktop – so if files are big it needs to be on a fast connection!

Connecting to your storage

You need the name and key for the storage.  This is available under your storage area in Azure.  Pick your storage container and then at the bottom there is a “Manage Access Keys” option.  You can use either primary or secondary key.

Managing snapshots

You can snapshot (or promote an old snapshot) using this tool

Downloading a vhd BLOB

If you download a blob VHD – this can be mounted locally.

It works on Hyper-V 2012 and on Windows 8 but NOT on 2008 R2 (and presumably Hyper V and Windows 7)

1. setting up and accessing SQL Azure databases
2. using SQL Azure with ArcGIS + ArcSquirrel and MapLink
3. using SQL Azure with Quantum GIS and MapServer.

Azure Management Portal

Although I’ve never set up any cloud services before (other than Arvixe web hosting) it was incredibly easy to setup a trial SQL Azure subscription (“Windows Azure Platform Introductory Special”), create a database, and add some firewall rules to control access. I didn’t dig deep, but the “Management Portal” seems nice and simple:

It helpfully tells you your connection string for ADO.Net, ODBC and PHP (passwords not included!):

See this page for details of the Windows Azure Platform Introductory Special including what you get for no charge and the pricing for the rest.

MapLink, ArcSquirrel and Azure

Hartmut connected up our MapLink to the database and pushed some spatial data into my Azure database from ArcGIS – 20,000 polygons took about 75 minutes.  Hartmut had the layers drawing in ArcMAP, performing as well as in a normal SQL Server instance apparently, with only minor mods to our code to make the initial connection. However editing will not work without some more tweaks - MapLink and ArcSquirrel rely on some SQL metadata that is a little different on Azure.

SQL Server Management Studio and MS Access

Connecting with SSMS is simple, however you get reduced functionality – for example you cannot right-click > edit top 200 records or anything, so quick edits of a small table need to be done with SQL or in MS Access or some other client. Fortunately connecting with MS Access (ODBC + SQL Native Client 10) is also simple, and  the Azure database behaves just like a local SQL instance – remarkable querying speed (all things considered), though slower inserts (10,000 records took something like 15 minutes). NB you cannot directly link a spatial table into Access because it doesn’t recognize the geometry data type; but to update attributes you can link a view.

Also, in SSMS you get no designer tools, e.g. for creating a view; everything has to be done with T-SQL.

SQL Azure Database Manager

After writing the above I found the database manager in the Azure control panel, which has functionality similar to SSMS, including the ability to edit the tables*:

* spatial data types not supported for view or edit.

You get a designer for creating/modifying tables, but views and stored procedures are T-SQL only.

MapServer, OGR and Azure

Next I wanted to try serving the data through MapServer, and viewing/editing the data in Quantum GIS or MapInfo.  MapServer and QGIS require the “geometry_columns” and “spatial_ref_sys” tables, and the easiest way to create these is to push some data in with ogr2ogr.

So I tried this from one of our London servers. I loaded a few hundred linear features, but got four errors like “INSERT command for new feature failed… incorrect syntax…”. On checking the results, a) while the geometries had loaded fine, all the attribute fields were NULL, and b) the spatial_ref_sys table was empty.  I had four attribute fields, so my guess is that each syntax error was one field update across all records using a join on the incoming data.

Next I hooked up a MapServer WMS to my SQL Azure tables. MapServer had no trouble connecting to my Azure tables, but initially wouldn’t draw anything…  on investigating I found that ogr2ogr had not correctly set the SRID of the geometries, so I had to update them “manually” with:

update esdm_waters set ogr_geometry.STSrid = 4326

(which, incidentally, took 1 minute 52 seconds for 788 rows, demonstrating that updating an Azure database ain’t quick).

Clearly OGR needs a little work to do on SQL Azure support, but this was enough to get me going.

The WMS worked, so to challenge it a little more I set up another WMS onto a table with about 8000 sub-tidal habitat polygons, showing red in this OpenLayers map:

And here showing the rivers WMS as well (blue lines):

Performance of these two layers was perfectly acceptable, though actually about 100% slower than the same WMS working from a local SQL Server 2008 Express instance. It would be interesting to see what it could do with MapServer also running on Azure.

I also tried a WFS – again, performance was acceptable though significantly slower than running with a local SQL instance (though I didn’t do strict comparisons).

Quantum GIS and SQL Azure

I found this really helpful blog post to get started: http://woostuff.wordpress.com/2011/03/13/opening-ms-sql-spatial-in-qgis/

Adding the layers requires some python script, but I’m sure someone will write a GUI plugin for QGIS soon.

First make sure you are running a version of QGIS that has GDAL/OGR 1.8 libraries – I’m on 1.7.0 which is OK.

Open the Plugins menu > Python Console

and type in these two commands (modified for your data):

uri = "MSSQL:server={serverName};database={databaseName};tables={tableName};trusted_connection=yes"
qgis.utils.iface.addVectorLayer(uri,'{yourLayerNameHere}','ogr')

In my case:

uri = "MSSQL:server=tcp:a0awtn7gg9.database.windows.net;database=ArcSquirrel1;tables=esdm_waters;Uid=exegesis@a0awtn7gg9;Pwd=myPassword;"
qgis.utils.iface.addVectorLayer(uri,'Rivers','ogr')

Without any fuss my rivers dataset appeared:

And I can create and edit features and their attributes, here adding a test line off the Hebrides:

And to prove it, here’s my new feature in OpenLayers…

And finally, here we have the habitat polygons (EPSG:27700) colour coded by habitat type, as well as the rivers (EPSG:4326), in QGIS. Polygon editing also worked fine, once I’d worked out why my new polygons kept vanishing (doh, classified symbols).

Lots to think about, and I look forward to editing the same data with ArcGIS and ArcSquirrel soon…