Ajeet Singh Raina

Building Your First Certified Kubernetes Cluster On-Premises, Part 1

Ajeet Singh Raina

This is the first in a series of guest blog posts by Docker Captain Ajeet Raina diving in to how to run Kubernetes on Docker Enterprise. You can follow Ajeet on Twitter @ajeetsraina and read his blog at http://www.collabnix.com

There are now a number of options for running certified Kubernetes in the cloud. But let’s say you’re looking to adopt and operationalize Kubernetes for production workloads on-premises. What then? For an on-premises certified Kubernetes distribution, you need an enterprise container platform that allows you to leverage your existing team and processes. 

Enter Docker Kubernetes Service

At DockerCon 2019, Docker announced the Docker Kubernetes Service (DKS). It is a certified Kubernetes distribution that is included with Docker Enterprise 3.0 and is designed to solve this fundamental challenge.

In this blog series, I’ll explain Kubernetes support and capabilities under Docker Enterprise 3.0, covering these topics:

  1. Deploying certified Kubernetes Cluster using Docker Enterprise 3.0 running on a Bare Metal System
  2. Implementing Persistent storage for Kubernetes workload using iSCSI
  3. Implementing Cluster Ingress for Kubernetes
  4. Deploying Istio Service Mesh under Docker Enterprise 3.0
  5. Support of Kubernetes on Windows Server 2019 with Docker Enterprise 3.0

So About DKS…

DKS is the only offering that integrates Kubernetes from the developer desktop to production servers, with ‘sensible secure defaults’ out-of-the-box. Simply put, DKS makes Kubernetes easy to use and more secure for the entire organization. Here are three things that DKS does to simplify and accelerate Kubernetes adoption for the enterprise:

  • Consistent, seamless Kubernetes experience for developers and operators. With the use of Version Packs, developers’ Kubernetes environments running in Docker Desktop Enterprise stay in sync with production environments for a complete, seamless Kubernetes experience. 
  • Streamlined Kubernetes lifecycle management (Day 1 and Day 2 operations). A new Cluster Management CLI Plugin allows operations teams to easily deploy, scale, backup and restore and upgrade a certified Kubernetes environment using a set of simple CLI commands.
  • Enhanced security with ‘sensible defaults.’ Teams get out-of-the-box configurations for security, encryption, access control, and lifecycle management — all without having to become Kubernetes experts.

DKS is compatible with Kubernetes YAML, Helm charts, and the Docker Compose tool for creating multi-container applications. It also provides an automated way to install and configure Kubernetes applications across hybrid and multi-cloud deployments. Capabilities include security, access control, and lifecycle management. Additionally, it uses Docker Swarm Mode to orchestrate Docker containers.

Kubernetes 1.14+ in Docker Enterprise

Docker Enterprise 3.0 comes with the following components:

  • Containerd 1.2.6
  • Docker Engine 19.03.1
  • Runc 1.0.0-rc8
  • docker-init 0.18.0
  • Universal Control Plane 3.2.0
  • Docker Trusted Registry 2.7
  • Kubernetes 1.14+
  • Calico v3.5.7
Docker UCP manager and worker nodes.

In this first post of the series, I will show you how to deploy a Certified Kubernetes cluster using Docker Enterprise 3.0 on bare metal (meaning you can deploy on-premises).

Pre-Requisites:

  • Ubuntu 18.04 (at least 2 Node to setup Multi-Node Cluster)
  • Minimal 4GB RAM is required for UCP 3.2.0
  • Go to https://hub.docker.com/my-content.
  • Click the Setup button for Docker Enterprise Edition for Ubuntu.
  • Copy the URL from the field labeled Copy and paste this URL to download your Edition.

Now you’re ready to start installing Docker Enterprise and Kubernetes. Let’s get started.

Step 1: Install packages to allow apt to use a repository over HTTPS

$sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

Step 2: Add the $DOCKER_EE_URL variable into your environment

Replace with the URL you noted down in the prerequisites. Replace sub-xxx too.

$curl -fsSL https://storebits.docker.com/ee/m/sub-XXX-44fb-XXX-b6bf-XXXXXX/ubuntu/gpg | 
sudo apt-key add -

Step 3: Add the stable Repository

$sudo add-apt-repository \
   "deb [arch=amd64] https://storebits.docker.com/ee/m/sub-XXX-44fb-XXX-b6bf-XXXXXX/ubuntu \
   $(lsb_release -cs) \
   stable-19.03"

Step 4: Install Docker Enterprise

$sudo apt-get install docker-ee docker-ee-cli containerd.io

Step 5: Verifying Docker Enterprise Version

$ sudo docker version
Client: Docker Engine - Enterprise
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        f660560
 Built:             Thu Jul 25 20:59:23 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       f660560
  Built:            Thu Jul 25 20:57:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
cse@ubuntu1804-1:~$

Step 6: Test the Hello World Example

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:6540fc08ee6e6b7b63468dc3317e3303aae178cb8a45ed3123180328bcc1d20f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the executable
    that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it to your
    terminal.

To try something more ambitious, you can run an Ubuntu container with:

 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID by signing up or logging in to Docker Hub.

For more examples and ideas, visit the Docker Docs getting started page.

Step 7: Install Universal Control Plane v3.2.0

$ sudo docker container run --rm -it --name ucp \
>   -v /var/run/docker.sock:/var/run/docker.sock \
>   docker/ucp:3.2.0 install \
>   --host-address 10.94.214.115 \
>   --interactive

Step 8: Accessing the UCP

Now you should be able to access Docker Universal Control Plane via https://<node-ip>

Click on “Sign In” and upload the license file to access Docker Enterprise UCP 3.2.0 WebUI as shown below:

Step 9: Adding Worker Nodes to the Cluster

Let’s add worker nodes to the cluster. Click on “Shared Resources” on the left pane and Click on “Nodes”. Select “Add Nodes” and choose an orchestrator. You can also add either Linux or Windows nodes to the cluster here as shown below:

I assume that you have a worker node installed with Ubuntu 18.04 and the latest Docker binaries (it can be either the free version of Docker Engine or Docker Enterprise).

@ubuntu1804-1:~$ sudo curl -sSL https://get.docker.com/ | sh

$ sudo usermod -aG docker cs
$ sudo docker swarm join --token SWMTKN-1-3n4mwkzhXXXXXXt2hip0wonqagmjtos-bch9ezkt5kiroz6jncid
rz13x <managernodeip>:2377
This node joined a swarm as a worker.

By now, you should be able to see both manager node and 1 worker node added under UCP.

If you see a warning on the UCP dashboard stating that you have a similar hostname on both the manager and worker node, change the hostname on the worker node and it will automatically get updated on UCP dashboard.

Step 10: Install the Docker Client Bundle

Click on Dashboard and scroll down to see the Docker CLI option. This option allows you to download a client bundle to create and manage services using the Docker CLI client. Once you click, you will be able to find a new window as shown below:

Click on “user profile page” and it should redirect you to https://<manager-ip-node/manage/profile/clientbundle page as seen in the below screenshot:

Click on “Generate Client Bundle” and it will download ucp-bundle-<username>.zip

$ unzip ucp-bundle-ajeetraina.zip
Archive:  ucp-bundle-ajeetraina.zip
 extracting: ca.pem
 extracting: cert.pem
 extracting: key.pem
 extracting: cert.pub
 extracting: kube.yml
 extracting: env.sh
 extracting: env.ps1
 extracting: env.cmd
 extracting: meta.json
 extracting: tls/docker/key.pem
 extracting: tls/kubernetes/ca.pem
 extracting: tls/kubernetes/cert.pem
 extracting: tls/kubernetes/key.pem
 extracting: tls/docker/ca.pem
 extracting: tls/docker/cert.pem
@ubuntu1804-1:~$ eval "$(<env.sh)"

The env script updates the DOCKER_HOST and DOCKER_CERT_PATH environment variables to make the Docker CLI client interact with UCP and use the client certificates you downloaded. From now on, when you use the Docker CLI client, it includes your user specific client certificates as part of the request to UCP.

Step 11: Install Kubectl on Docker Enterprise 3.0

Once you have the Kubernetes version, install the kubectl client for the relevant operating system. As shown below, we need to install Kubectl version 1.14.3:

Step 12: Set the Kubectl version

@ubuntu1804-1:~$ k8sversion=v1.14.3
@ubuntu1804-1:~$ curl -LO https://storage.googleapis.com/kubernetes-release/release/
$k8sversion/bin/linux/amd64/kubectl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 41.1M  100 41.1M    0     0  7494k      0  0:00:05  0:00:05 --:--:-- 9070k
@ubuntu1804-1:~$ chmod +x ./kubectl
@ubuntu1804-1:~$ sudo mv ./kubectl /usr/local/bin/kubectl
@ubuntu1804-1:~$

Step 13: Verify the Kubectl Installation

~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:
"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:
"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.3-docker-2", 
GitCommit:"7cfcb52617bf94c36953159ee9a2bf14c7fcc7ba", GitTreeState:"clean",
BuildDate:"2019-06-06T16:18:13Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"

Step 14: List out the Kubernetes Nodes

cse@ubuntu1804-1:~$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
node2          Ready    <none>   23h   v1.14.3-docker-2
ubuntu1804-1   Ready    master   23h   v1.14.3-docker-2

Step 15: Enabling Helm and Tiller with UCP

$ kubectl create rolebinding default-view --clusterrole=view --serviceaccount=kube-system
:default --namespace=kube-system
rolebinding.rbac.authorization.k8s.io/default-view created

$ kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin 
--serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/add-on-cluster-admin created
cse@ubuntu1804-1:~$

Step 16: Install Helm

$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7001  100  7001    0     0   6341      0  0:00:01  0:00:01 --:--:--  6347
$ chmod u+x install-helm.sh

$ ./install-helm.sh
Downloading https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/local/bin
helm installed into /usr/local/bin/helm
tiller installed into /usr/local/bin/tiller
Run 'helm init' to configure helm.

cse@ubuntu1804-1:~$ helm init
Creating /home/cse/.helm
Creating /home/cse/.helm/repository
Creating /home/cse/.helm/repository/cache
Creating /home/cse/.helm/repository/local
Creating /home/cse/.helm/plugins
Creating /home/cse/.helm/starters
Creating /home/cse/.helm/cache/archive
Creating /home/cse/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/cse/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users'
policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/
#securing-your-helm-installation
cse@ubuntu1804-1:~$

Step 17: Verify the Helm Installation

$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085"
, GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085"
, GitTreeState:"clean"}

Step 18: Deploying MySQL using Helm on Docker Enterprise 3.0

Let’s try out deploying MySQL using HelmPack.

$ helm install --name mysql stable/mysql
NAME:   mysql
LAST DEPLOYED: Wed Aug  7 11:43:01 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME        DATA  AGE
mysql-test  1     0s

==> v1/PersistentVolumeClaim
NAME   STATUS   VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE
mysql  Pending  0s

==> v1/Secret
NAME   TYPE    DATA  AGE
mysql  Opaque  2     0s

==> v1/Service
NAME   TYPE       CLUSTER-IP   EXTERNAL-IP  PORT(S)   AGE
mysql  ClusterIP  10.96.77.83  <none>       3306/TCP  0s

==> v1beta1/Deployment
NAME   READY  UP-TO-DATE  AVAILABLE  AGE
mysql  0/1    0           0          0s


NOTES:
MySQL can be accessed via port 3306 on the following DNS name from within your cluster:
mysql.default.svc.cluster.local

To get your root password run:

    MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql -o jsonpath=
"{.data.mysql-root-password}" | base64 --decode; echo)

To connect to your database:

1. Run an Ubuntu pod that you can use as a client:

    kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il

2. Install the mysql client:

    $ apt-get update && apt-get install mysql-client -y

3. Connect using the mysql cli, then provide your password:
    $ mysql -h mysql -p

To connect to your database directly from outside the K8s cluster:
    MYSQL_HOST=127.0.0.1
    MYSQL_PORT=3306

    # Execute the following command to route the connection:
    kubectl port-forward svc/mysql 3306

    mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}


cse@ubuntu1804-1:~$

Step 19: Listing out the Releases

The helm list command lists all of the releases. By default, it lists only releases that are deployed or failed. Flags like ‘–deleted’ and ‘–all’ will alter this behavior. Such flags can be combined: ‘–deleted –failed’. By default, items are sorted alphabetically. Use the ‘-d’ flag to sort by release date.

$ helm list
NAME    REVISION      UPDATED                      STATUS         CHART          APP VERSION     NAMESPACE
mysql   1             Wed Aug  7 11:43:01 2019     DEPLOYED       mysql-1.3.0    5.7.14          default

$ kubectl get po,deploy,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/mysql-6f6bff58d8-t2kwm   1/1     Running   0          5m35s

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/mysql   1/1     1            0           5m35s

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP    28h
service/mysql        ClusterIP   10.96.77.83   <none>        3306/TCP   5m35s
cse@ubuntu1804-1:~$

With DKS, you can use Helm flawlessly with UCP under Docker Enterprise 3.0.

Kubernetes, On-Premises

Now you have Kubernetes running on-premises. You can do a lot from here, and I’ll cover some possibilities in the rest of this series.

You may also want to experiment with designing your first application in Kubernetes. Bill Mills from the Docker training team wrote a great blog series recently covering just that. I highly recommend checking it out starting with part 1 here.

Have a look at these resources if you’re looking to learn more about Docker Enterprise 3.0 and Kubernetes: