Deployment at last

While my current contract doesn't leave much time for personal projects, I have made some progress on my current project (details on exactly what this is to follow). In fact, some of the smaller, peripheral services have their primary use-cases functionally complete and are ready for deployment and I am now faced with the question: Deployment to where?

Fabric or Container

Given this project constitutes multiple micro-services using message based, asynchronous communication with the potential to scale services horizontally, I required some form of elastic service fabric. Furthermore, I wanted a local development environment which would simulate a cluster of machines but with which I could monkey about as much as I liked without fear of accidentally incurring massive hosting fees in the cloud.

As I had just upgraded my home server with plenty of memory, I decided to use one of more virtual machines running on this server to host the environment during development, but which technology to use?

Initially I had intended to use a local Service Fabric cluster. However, upon further investigation I found that the SDK and API introduced significant friction to the development process (needing additional projects for supplying manifest / configuration data for services, overly complex deployment scripts, etc). Even the 'guest executable' approach seemed overly complex and I quickly went off this approach.

My second thought was Docker; specifically the creation of a local Docker Swarm which I could deploy servies to with Docker Compose. Docker Machine made short work of provisioning Docker hosts in Hyper-V but with one caveat: it's boot2docker image would only run Linux based containers and, while many of the services I have written / will write run quite happily on .NET Core, some require packages that do not yet provide support for .NET Core / Standard.

Given that a recent update made Swarm mode available to Windows Server 2016 host operating systems, I decided I would look into provisioning a series of Windows Server VM's with container support and configure Docker on these VM's to operate in swarm mode.

Using Nano Server as a Docker host

While I had previously used Microsoft Nano Server as a guest OS for containerized apps, I hadn't realised that it was possible to use it as a host OS for Docker until I came across this article. For those not familiar with Nano Server it is an extremely slimmed down (the OS image is less than 170Mb) and fast booting (5-10 seconds), headless version of Windows Server 2016 which, given it is capable of acting as a Docker host, effectively makes it 'boot2docker' but for Windows containers.

Nano Server is shipped with Windows Server 2016 and is accompanied by a Powershell module which provides some terrific facilities for working with Nano Server images. This document shows how to use this Powershell module to create customised Nano Server images as either '.wim', '.iso' or - most interestingly for me - '.vhdx'. In short, the following powershell command will create a virtual HD that can be attached to a virtual machine and which will boot directly into Nano Server with support for - but no utilites to provide - containerization services:

New-NanoServerImage -Edition Standard -DeploymentType Guest -MediaPath <path to root of media> -BasePath <path in which to build the image> -TargetPath <destination path>\NanoServer.vhdx -ComputerName <computer name> -Containers -EnableRemoteManagementPort

This command will also open WinRM ports on the Nano Server which allows you to use PS Remoting to remote into the virtual machine and examine it's state; indispensable for debugging purposes.

Updating Nano Server

Referring back to the 'Getting Started with Swarm Mode' article, it states that a relatively recent update is required to run Docker Swarms on Windows Server 2016 based OS's. It is therefore necessary to ensure this update is applied to any NanoServer image we create for the purpose of running Docker, ideally during the creation of the image not a subsequent setup script.

Well, those clever people at Microsoft thought of this too and provided a -ServicingPackagePath argument for the New-NanoServerImage command which takes a path to a cab update file and applies the update to the NanoServer OS during image creation. A mechanism for getting an update from Microsoft (as an *.msu) and extracting it into a (series of) cab-files for use in the New-NanoServerImage is provided by Thomas Maurer in an excellent blog post here.

Installing Docker as part of a Nano Server image

Now, just like 'boot2docker' we want our Nano Server to be ready to host Windows Containers as soon as it's booted and without further manual configuration. To this end, I needed to find a way to install Docker as part of the deployment process. Fortunately the 'New-NanoServer' command provides a -SetupCompleteCommand argument which allows you to 'run custom commands as part of setupcomplete.cmd' (i.e. on first boot). Great, so now to prepare a script to deploy Docker which we can call via the -SetupCompleteCommand argument.

Conveniently, Docker's documentation for installing Docker EE (the version supported by Windows) provides exactly the script required, copied below with some additional configuration copied from the 'Prepare Container Host' section of 'Deploy Containers on Nano' article:

# On an online machine, download the zip file
Invoke-Webrequest -UseBasicparsing -Outfile docker.zip https://download.docker.com/components/engine/windows-server/17.03/docker-17.03.0-ee.zip

# Extract the archive.
Expand-Archive docker.zip -DestinationPath $Env:ProgramFiles

# Clean up the zip file.
Remove-Item -Force docker.zip

# Install Docker. This will require rebooting.
# This is not required as we have already prepared out image with container support
# $null = Install-WindowsFeature containers

# Add Docker to the path for the current session.
$env:path += ";$env:ProgramFiles\docker"

# Modify PATH to persist across sessions.
# Note: Nano Server's SetEnvironmentVariable method does not take a scope parameter 
[Environment]::SetEnvironmentVariable("PATH", $env:path)

# Open an inbound port for the docker daemon  
netsh advfirewall firewall add rule name="Docker daemon " dir=in action=allow protocol=TCP localport=2375

# Create and populate docker daemon's configuration file
New-Item -Type File 'C:\ProgramData\docker\config\daemon.json' -Force
Add-Content 'C:\programdata\docker\config\daemon.json' '{ "hosts": ["tcp://0.0.0.0:2375", "npipe://"] }'

# Register the Docker daemon as a service.
dockerd --register-service

# Start the daemon.
Start-Service docker

Now this script requires that the Nano Server be online when the script is run so that it can download the Docker binaries. Unfortunately, given that this script will run as part of the deployment process, this is unlikely to be the case. Instead, we'll need to lean on another feature of the New-NanoServerImage, -CopyPath. This argument allows you to specify one or more files to copy to the Nano Server image as part of it's creation and we'll use it to copy a pre-downloaded copy of the docker binaries (along with a copy of the deployment script) to the root of the images C:\ drive, as shown here:

New-NanoServerImage -Edition Standard -DeploymentType Guest -MediaPath <path to root of media> -BasePath <path in which to build the image> -TargetPath <destination path>\NanoServer.vhdx -ComputerName <computer name> -Containers -EnableRemoteManagementPort -CopyPath @('<path to deployment script>\DeployDocker.ps1', '<path in which docker is downloaded>\docker.zip') -SetupCompleteCommand @('Powershell.exe -Command .\DeployDocker.ps1') 

Great, now we can comment out the first line of the script above and it'll run fine, right? Unfortunately not. It seems that, at the stage of the boot process at which this script runs, powershell isn't quite ready to run powershell. Fortunately, others have encountered this issue before and Sergey Babkin provides this solution; copied below with customisation for our requirements:

set LOCALAPPDATA=%USERPROFILE%\AppData\Local
set PSExecutionPolicyPreference=Unrestricted
Powershell -Command C:\DeployDocker.ps1

Ok, so we'll add and deploy this batch file as 'DeployDocker.bat' and then execute this instead of the powershell script as the -SetupCompleteCommand, as shown here:

New-NanoServerImage -Edition Standard -DeploymentType Guest -MediaPath <path to root of media> -BasePath <path in which to build the image> -TargetPath <destination path>\NanoServer.vhdx -ComputerName <computer name> -Containers -EnableRemoteManagementPort -CopyPath @('<path to deployment batch file'\DeployDocker.bat', '<path to deployment script>\DeployDocker.ps1', '<path in which docker is downloaded>\docker.zip') -SetupCompleteCommand 'C:\DeployDocker.bat' 

And thats it. If you now create a virtual machine with the new 'NanoServer.vhdx' image as it's boot drive, you should find that, once it's booted you're able to communicate with the Docker daemon on the VM. For example:

docker -H <IP Address of VM> ps

Should return a (empty) list of containers present on the Nano Server host.

Scripting the creation of a VM

Now, jumping through all the hoops above each time we want to make a new VM to host docker would be arduous to say the least. As such, I put together a powershell module which is able to directly create VM's ready to host Docker containers in just a few steps. From a powershell command prompr, this can be done as follows:

1. Get the Nano2Docker powershell module

This module is available in my Docker repository on Github and can be downloaded directly using the following command:

Invoke-WebRequest -OutFile Nano2Docker.psm1 https://raw.githubusercontent.com/ibebbs/Docker/master/Nano2Docker/Nano2Docker.psm1

Then installed using:

Import-Module Nano2Docker.psm1

2. Prepare a base NanoServer image

The process of downloading docker and applying updates can be extremely slow. Therefore we first create a reusable base NanoServer image that has docker installed and updates applied. This is done using the command:

Initialize-Nano2DockerImage -MediaPath <Drive letter for Windows Server 2016> -BuildPath <Path to a build location>

For example, if your Windows Server 2016 media is mounted in the 'G' drive and you want to build the new Nano2Docker image in the 'C:\Nano2Docker' folder, you'd use the command:

Initialize-Nano2DockerImage -MediaPath G: -BuildPath "C:\Nano2Docker"

The command provides defaults download locations for docker and required updates but these can be overriden using the -DockerUrl and -UpdateUrl parameters repsectively. Furthermore, if you already have an '.msu' file downloaded you can save a lot of time by providing this to the command using the -UpdateFile parameter.

When you run this command, you will be prompted for Administrator credentials for the new Nano2Docker image. Either enter them when prompted or supply a SecureString value to the -Password argument (which is usually done using the Get-Credentials commandlet).

When this command completes - which can take a while - you will find a 'Nano2Docker.vhdx' file in the BuildPath directory.

3. Create a VM using the new NanoServer image

You can now use the New-Nano2Docker commandlet to quickly create new VM docker hosts. The command is used as follows:

New-Nano2Docker -MediaPath <Drive letter for Windows Server 2016> -ImagePath <Path to a previously created Nano2Docker.vhdx>

Using our previous example, the command would be:

New-Nano2Docker -MediaPath G: -ImagePath "C:\Nano2Docker\Nano2Docker.vhdx"

This command will copy the base image to the VM path (defaults to 'C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks'), provision a new, local, Hyper-V VM named 'Nano2Docker' with reasonable resources (4 cores / 2Gb of dynamic ram) and attached to the first available virtual switch. Obviously, each of these characteristics can be changed using the appropriate arguments - use Get-Help New-Nano2Docker to list the available arguments. You will again be asked to an Administrator password for the new VM which can be entered during creation or specified as a parameter to the commandlet.

Once the new VM has been created, it will be started and the script will wait for NanoServer to start correctly. Once started, the IP address of the new VM is displayed and it should be immediately possible to use docker to communicate with the docker daemon on the VM.

Scripting the deployment of a Docker Swarm

Given it was now trivial to create docker hosts, I wrote a final commandlet which leverages previous commandlets to provision an entire docker swarm. This is done using the New-Nano2DockerSwarm commandlet as follows:

New-Nano2DockerSwarm -MediaPath <Drive letter for Windows Server 2016> -ImagePath <Path to a previously created Nano2Docker.vhdx> -VMPath <Path to a location to store swarm host VMs>

Again, using our previous example and wanting to store new VM images in the 'C:\Nano2Docker\VMs' directory, the command would be:

New-Nano2DockerSwarm -MediaPath G: -ImagePath "C:\Nano2Docker\Nano2Docker.vhdx" -VMPath "C:\Nano2Docker\VMs"

This command defaults to creating a single manager node and three worker nodes named 'n2d-mngr-0' and 'n2d-wrkr-0/1/2' respectively. All nodes will have already been joined to the swarm in the appropriate capacity and ready for services to be deployed using docker or docker-compose. Again, the number of manager and worked nodes as well as the name prefix for each node can be changed via the appropriate arguments (use Get-Help New-Nano2DockerSwarm to list the arguments).

Roundup

I've successfully used this script to provision multiple docker hosts and docker swarms but it's worthwhile noting that it can be a bit finickity with directories. If you can an error saying it couldn't find a directory then simply create it first. I really ought to fix up the script (improvements via PR gratefully accepted) but I'm now a bit busy deploying services to my swarm :0)

Furthermore I've successfully used nodes created by this script to provision a hybrid swarm (see the 'Mixed OS clusters section here) of boot2docker and nano2docker images. With the appropriate labels on the nodes, it is possible to use docker-compose to automatically deploy images to the correct hosts while retaining all the benefits of overlay networking.

It's a lot of docking fun!