Building Docker Enterprise Cluster using Terraform

October 19, 2019 - 8 minutes read - 1701 words

In this article, I would like to show how you can use Terraform to fully automate the build of a 3-node Docker Enterprise 2.1 cluster. If you are in the process of planning or setting up an Enterprise Docker Cluster for your containers, in any of the Public Cloud platform, I hope you will find this article and the templates quite useful.

Building Docker Enterprise 2.1 Cluster using Terraform

Even if you are not building a Docker Enterprise cluster, you may still find this article useful if you’d like to automate your infrastructure build in Alibaba Cloud.

As my objective was to show the art of possible using IaC (Infrastructure as Code) to automate build in Alicloud, I thought why not do that using the most popular platforms, hence decided

to build a Containers-as-a-Service platform using Docker Enterprise Edition 2.1 AND
to build that platform using Terraform

Docker Enterprise Edition

Docker Enterprise 2.1 is a Containers-as-a-Service (CaaS) platform that enables a secure software supply chain and deploys diverse applications for high availability across disparate infrastructure, both on-premises and in the cloud. It is a secure, scalable, and supported container platform for building and orchestrating applications across multi-tenant Linux, Windows Server 2016, and IBM Z environments.

One thing that I always loved about Docker is their Simplicity and Customer Centricity. That’s exactly what they have done with the release of Enterprise 2.1 too. With Docker EE 2.1, you now have a freedom of choice as it can,

reliably support both Windows and Linux containers
be hosted in any cloud platform or on-premise Data Center
use both Docker Swarm and Kubernetes orchestration interchangeably

So, if you are an Enterprise customer who is looking to embark on a project to either migrate your legacy applications to Containers or keen to embrace DevOps for development of new applications, I’d strongly recommend Docker EE 2.1 as your CaaS platform.

You can start small and scale your cluster as you grow your container base. You can start utilizing the much simpler Docker Swarm for initial orchestration and switch to Kubernetes later if you really need it.

Docker EE 2.1 cluster also comes with components such as,

Docker UCP - which gives a single-pane-of-glass across your cluster
Docker Trusted Registry - to securely host your container images

It also has enterprise security features like encrypted communication, application isolation, vulnerability scanning for images and much more.

Terraform

This has to be my favorite Orchestration/IaC tool out there. I just love the power and flexibility that Terraform offers for deploying new services to any Public Cloud platform. You just define what you need and ask Terraform to Go and Build. It is that simple.

I chose Terraform for this automation as it is pretty much platform agnostic. Though you can’t use the same templates for any Cloud service provider it is quite easy to customize to a different provider once it is developed for a specific Cloud platform.

To learn more about Terraform, visit the HashiCorp website or read this good summary About Terraform in Alberto Roura’s article in Alibaba Cloud Techshare.

Key references

This section lists number of useful references if you would like to learn more about Docker, Terraform, Automation

Docker Enterprise https://docs.docker.com/ee/ https://www.docker.com/products/docker-enterprise
Terraform https://www.terraform.io https://www.terraform.io/docs/providers/alicloud/index.html
My GitHub repository https://github.com/sajiv3m/docker-terraform-alicloud

Building the Docker Enterprise Cluster

For this demo, I chose to build a small 3-node Docker Enterprise 2.1 cluster.

One Linux server that hosts both Docker UCP and Docker Trusted Registry (DTR). Same node will also be configured as the Docker Swarm Manager and Kubernetes Master.
One Linux host which will be automatically joined as a worker node, in the Docker Swarm created by the UCP host.
One Windows host which will be automatically joined as a worker node, in the Docker Swarm created by the UCP host.

If you would like to get on with the cluster build right away, go to my GitHub repository and follow the instructions there.

Once you get the pre-requisites ready, you could get the cluster built in less than 30 minutes.

Terraform files explained

If you rather understand the templates first, before jumping into action, following sections will take you through the details.

The terraform configuration files in my GitHub repository, and the bash & powershell scripts, have adequate comments to help you understand them better. In this article, I would like to focus on explaining the purpose of each of those template or script.

Terraform, as you may be aware, helps to deploy the Infrastructure by letting users define what they need, using the High-level Configuration Language (HCL). Terraform will build a detailed execution plan based on this definition and deploy the infrastructure on the platform specified.

Terraform supports a wide range of Cloud service providers and certainly the major Cloud providers and VMWare vSphere for on-premise Data Center.

The complete definition of our Docker Enterprise Cluster could be specified in a single .tf definition file but it is recommended to split into multiple .tf files for ease of maintaining and readability. All .tf files in the main folder form part of the root (main) module.

Terraform will load all configuration files (.tf files) in the folder, in alphabetical order. The order of variables, resources, etc. defined within the configuration doesn’t matter. Terraform configurations are declarative, so references to other resources and variables do not depend on the order they’re defined.

Variables

Input variables serve as parameters for the Terraform module. All variables must be declared in a .tf file (e.g., variables.tf) and their values could be passed in command line during execution OR in a separate .tfvars file (e.g., terraform.tfvars)

I recommend using UPPER CASE for the input variable names so they stand out from the terraform local variables, defined using locals and terraform resource names. I would also suggest to give the values for variables in a separate terraform.tfvars file, so you can choose to not send this file when sharing with others as it may contain your account secret keys, passwords etc. During execution, if terraform cannot find a .tfvars file and the default values for variables are not defined, it prompts the user.

Provider

provider.tf This file defines the keys to connect to Alicloud and the region where you want the resources created

Network and Security

network-security.tf This file defines the VPC, vSwitch, Security Group and the security/firewall rules to restrict access to the docker hosts.

The vSwitch and Security groups defined in this file will be mapped to our hosts, in the ECS (Elastic Compute Service) definition.

Few points to note,

Priority value for the security roles range from 1-100. Smaller the value, higher the priority
RDP, WinRM and SSH access are allowed only from a specific IP - your Public IP.
Kubernetes, Docker and application access are allowed from anywhere

Compute for Docker UCP Manager

docker-host.tf

This file defines the main host where Docker UCP and DTR will be installed. As part of Docker UCP installation, it will also initiate creation of Docker Swarm to which the other hosts will join as worker nodes.

Few points to note,

internet_max_bandwidth_out attribute ensures a Public IP is assigned as part of VM creation.
This Public IP is used in the connection definition for file provisioner to copy the installation scripts and for remote-exec provisioner to run the setup automatically for Docker EE, UCP and DTR
ssh key pair for the UCP host and the linux worker node is defined in key-pair.tf

Compute for Docker Worker nodes

linux-worker.tf This file defines the linux worker node. Quite similar to the docker-host.tf file, except that this only has scripts to install Docker EE and join as worker node.

windows-worker.tf This file defines the windows worker node. Key differences from the linux node definition are

windows image instead of the linux image
password attribute instead of the key_name attribute used in Linux for ssh-key pair.
user-data attribute which is used for bootstrapping the VM, including configuration of WinRM, so further scripts can be executed using WinRM
connection type is winRM instead of ssh used in Linux host

Output

output.tf This file defines the values returned by the module. These values will be printed once Terraform successfully completes the execution.

Values defined in output.tf can be printed anytime using the command terraform output, after a successful execution. To retrieve further details of the Terraform execution, use terraform show

Scripts to automate build of Docker components

Scripts used to bootstrap Linux hosts are in the folder lin-files. They are called using the remote-exec provisioner in the ECS VM definition.

These scripts are used to setup Docker Enterprise, Docker UCP and Docker Trusted Registry as per the documentation in Docker website.

Scripts used to bootstrap Windows host are in the folder win-files. They are called using the remote-exec provisioner in the ECS VM definition.

These scripts are used to setup Docker Enterprise and also to join as worker node. Docker UCP and Docker Trusted Registry cannot be installed on Windows hosts. They can only be worker nodes.

Key point to note in the Windows scripts,

To ensure Invoke-RestMethod works fine when accessing the Docker API, following changes had to be done in the script prior to calling the Docker API.

Change the default TLS version in powershell from 1.0. to 1.2
Ignore the certificate errors when connecting to Docker API

Conclusion

This article was intended only to introduce you to the basic concepts of Docker Enterprise and how you can use Terraform to automate deployments in Alicloud. The definition files and scripts in the GitHub repository will help you setup a basic Docker Enterprise cluster and also provide you tips which you can use for other installations.

In the subsequent articles, I will share information on how to scale your cluster with a load balancer and additional nodes and also details on deploying containers using Docker Swarm or Kubernetes orchestration.