How to set up a Digital Ocean Kubernetes cluster (Q1 2019).

Introduction

Overview of how a request gets from a browser to a Docker container managed by Kubernetes:

Internet[Browser => DigitalOcean LoadBalancer] => Kubernetes[Ingress => Service => Deployment => Pod] => Docker[Container]

• https://kubernetes.io/docs/concepts/services-networking/ingress/
• https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
• https://kubernetes.io/docs/concepts/services-networking/service/
• https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
• https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
• https://kubernetes.io/docs/concepts/workloads/pods/pod/

Prerequisites

• Install kubectl

$ brew install kubectl

1. Create cluster in DigitalOcean dashboard

Download the cluster configuration (YAML file) and put it in the ~/.kube folder.

1. Set KUBECONFIG environment variable

$ export KUBECONFIG=/Users/christian/.kube/staging-kubeconfig.yaml
$ kubectl config current-context
$ kubectl get nodes
NAME               STATUS   ROLES    AGE   VERSION
xxx-staging-qv72   Ready    <none>   70m   v1.14.1
xxx-staging-qv7p   Ready    <none>   71m   v1.14.1
xxx-staging-qv7s   Ready    <none>   71m   v1.14.1

1. Install ingress controller

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/cloud-generic.yaml

Get the EXTERNAL-IP of the load balancer:

$ kubectl get services --namespace=ingress-nginx

1. Create ingress resource

ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
        - path: /
          backend:
            serviceName: app-service
            servicePort: 80

$ kubectl apply -f ingress.yml

1. Create deployment

deployment.yaml:

apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  ports:
  - port: 80
    targetPort: 5678
  selector:
    app: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app: app
  replicas: 1
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: hashicorp/http-echo
        args:
        - "-text=Hello. This is your Digital Ocean k8s cluster."
        ports:
        - containerPort: 5678

$ kubectl apply -f deployment.yml

1. Curl your app

curl https://<EXTERNAL-IP> --insecure

References

• https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes?comment=76348#step-2-%E2%80%94-setting-up-the-kubernetes-nginx-ingress-controller

One day, you see CrashLoopBackOff in the kubectl output:

$ kubectl get pod
NAME                               READY   STATUS             RESTARTS   AGE
app-548c9ddc46-z2fng               0/1     CrashLoopBackOff   79         6h26m

You already know that executing bash in the container is not possible because the container has crashed:

$ kubectl exec -ti app-548c9ddc46-z2fng bash
error: unable to upgrade connection: container not found ("app")

Option 1: Analyze the container logs

You can view the container’s logs with kubectl logs:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 2: Modify Dockerfile’s CMD

Modifying the Dockerfile’s ‘CMD’ is not needed, as it can be done indirectly through the Pod’s YAML configuration file:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: company/app
      # DEBUG env variables
      command: [ "/bin/sh", "-c", "env" ]

The modified command will print the environment variables to the logs:

command: [ "/bin/sh", "-c", "env" ]

To view the output run:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 3: Start a shell and sleep (CMD)

Sleeping usually helps, and this can be done by modifying the container’s command:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: xxx-service
spec:
  replicas: 1
  template:
    ...
    spec:
      containers:
      - image: yyy/xxx:1.0.0
        name: xxx-service
        ...
        command:
          - "sh"
          - "-c"
          - "sleep 10000"

The container will start and run “sleep 10000” in a shell, giving you exactly 10000 seconds to debug the issue by connecting to the “sleeping” container:

$ kubectl exec -ti app-548c9ddc46-z2fng bash

How to configure Kubernetes to pull images from a private Docker registry:

• First configure Docker by following the steps outlined here:

https://snippets.aktagon.com/snippets/869-configure-docker-to-use-a-private-container-registry-using-a-self-signed-certificate

• Verify that the Docker configuration contains the authentication information

sudo cat ~/.docker/config.json
{
    "auths": {
        "<registry-server>": {
            "auth": "<hash>"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.09.4 (linux)"
    }
}

• Base64 encode the config.json file

cat ~/.docker/config.json | base64 -w0 > config.base64.json

• Create secret.yml and add the contents of config.base64.json to dockerconfigjson

apiVersion: v1
kind: Secret
metadata:
 name: registrypullsecret
data:
 .dockerconfigjson: <config.base64.json>
type: kubernetes.io/dockerconfigjson

• Import the secret into Kubernetes

kubectl create -f secret.yml && kubectl get secrets

• Test that the secret was imported into Kubernetes

kubectl get secrets

How to configure Docker to use a private container registry using a self-signed certificate.

Tested on Docker version 18.09.4, build d14af54.

• Copy self-signed certificate from the registry server to the docker server

On your laptop:

$ scp christian@registry-server://etc/ssl/certs/selfsigned.crt christian@docker-server://etc/ssl/certs/private-docker-registry.crt

• Restart docker daemon

On the docker server:

$ sudo service docker restart

• Login to the registry from the docker server

On the docker server:

sudo docker login -u christian registry-server
> WARNING! Your password will be stored unencrypted in /home/christian/.docker/config.json.

Tested April, 2019.

1. Download the Kubernetes configuration from the DigitalOcean dashboard

2. Move the Kubernetes configuration file to ~/.kube folder

$ mkdir $HOME/.kube/projectx/
$ mv ~/Downloads/projectx-staging-kubeconfig.yaml $HOME/.kube/projectx/staging-kubeconfig.yaml

1. List all contexts in the KUBECONFIG variable

$ export KUBECONFIG=$HOME/.kube/projectx/staging-kubeconfig.yaml:$HOME/.kube/projectx/production-kubeconfig.yaml

1. Install kubectl

$ brew install kubectl

1. List context and nodes

$ kubectl config get-contexts
$ kubectl get nodes
$ kubectl config use-context

Always remember to set the time zone session setting when using database/sql and timestamps, otherwise timestamps will use the database default:

db, err = sql.Open("postgres", "timezone=UTC")
if err != nil {
    log.Fatal(err)
}
defer db.Close()
var timezone string
err = db.QueryRow("SHOW timezone").Scan(&timezone)
if err != nil {
    log.Fatal(err)
}
log.Println("timezone:", timezone)

Running SET TIME ZONE ‘Africa/Casablanca’; once in a connection won’t work because a pool of connections is used.

require 'resolv'
require 'ipaddr'

class PrivateIP
  # https://en.wikipedia.org/wiki/Private_network#Private_IPv4_address_spaces
  IPV4_NETWORKS = %w[0.0.0.0/8
  10.0.0.0/8
  100.64.0.0/10
  127.0.0.0/8
  169.254.0.0/16
  172.16.0.0/12
  192.0.0.0/24
  192.0.0.0/29
  192.0.0.8/32
  192.0.0.9/32
  192.0.0.170/32
  192.0.0.171/32
  192.0.2.0/24
  192.31.196.0/24
  192.52.193.0/24
  192.88.99.0/24
  192.168.0.0/16
  192.175.48.0/24
  198.18.0.0/15
  198.51.100.0/24
  203.0.113.0/24
  240.0.0.0/4
  255.255.255.255/32
  224.0.0.0/24
  239.0.0.0/8].map { |cidr| IPAddr.new(cidr) }

  # https://en.wikipedia.org/wiki/Unique_local_address
  IPV6_NETWORKS = %w[fd00::/8
  fc00::/8
  0000:0000:0000:0000:0000:0000:0000:0000/64].map { |cidr| IPAddr.new(cidr) }

  class InvalidHost < ArgumentError; end

  # Examples:
  # private?('127.0.0.1') => true
  # private?('localhost') => true
  # private?('google.com') => false
  def self.private?(ip_or_host)
    address = IPAddr.new(ip_or_host)
    if address.ipv4?
      IPV4_NETWORKS.any? { |cidr| cidr.include?(ip_or_host) }
    elsif address.ipv6?
      IPV6_NETWORKS.any? { |cidr| cidr.include?(ip_or_host) }
    else
      false
    end
  rescue IPAddr::InvalidAddressError
    private_host?(ip_or_host)
  end

  # Example: private_host?('localhost') => true
  def self.private_host?(host)
    host_ips(host).any? do |type, ips|
      ips.any? { |ip| private?(ip) }
    end
  end

  # Example: host_ips('localhost') => {ipv4: ['127.0.0.1'], ipv6: []}
  def self.host_ips(host)
    ipv4 = Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A)
    ipv6 = Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
    raise InvalidHost, "unknown host: #{host}" if ipv4.empty? && ipv6.empty?
    { ipv4: ipv4.map { |r| r.address.to_s },
      ipv6: ipv6.map { |r| r.address.to_s } }
  end
end

require 'test_helper'
require 'private_ip'

class PrivateIPTest < ActiveSupport::TestCase
  test "private?" do
    assert PrivateIP.private?('localhost')
    assert PrivateIP.private?('127.0.0.1')
    assert PrivateIP.private?('0.0.0.0')
    refute PrivateIP.private?('google.com')
    refute PrivateIP.private?('209.216.230.240')
  end

  test "private? (IPV6)" do
    assert PrivateIP.private?('fd7b:5886:20a0:11a0:1111:2222:3333:4444')
    assert PrivateIP.private?('::1') # Try http://[::1]:3000 in browser
    assert PrivateIP.private?('::')
    assert PrivateIP.private?('0:0:0:0:0:0:0:1')
  end

  test "private? (CIDR)" do
    PrivateIP::IPV4_NETWORKS.each do |cidr|
      assert PrivateIP.private?(cidr.to_s)
    end
    PrivateIP::IPV6_NETWORKS.each do |cidr|
      assert PrivateIP.private?(cidr.to_s)
    end
  end

  test "private_host?" do
    assert PrivateIP.private_host?('localhost')
    refute PrivateIP.private_host?('google.com')
  end

  test "host_ips" do
    assert_equal({ ipv4: ["127.0.0.1"], ipv6: [] }, PrivateIP.host_ips('localhost'))
    assert_equal({ ipv4: ["209.216.230.240"], ipv6: []}, PrivateIP.host_ips('news.ycombinator.com'))
  end

  test "host_ips (invalid)" do
    assert_raise PrivateIP::InvalidHost do
      PrivateIP.host_ips('https://google.com')
    end
    assert_raise PrivateIP::InvalidHost do
      PrivateIP.host_ips('127.0.0.1')
    end
    assert_raise PrivateIP::InvalidHost do
      PrivateIP.host_ips('::1')
    end
    assert_raise PrivateIP::InvalidHost do
      PrivateIP.host_ips('[::1]')
    end
    assert_raise PrivateIP::InvalidHost do
      PrivateIP.host_ips('::')
    end
  end
end

Access host from docker container IP:

# Mac
ping docker.for.mac.localhost

# Windows
ping docker.for.win.localhost

# Linux, see:
# https://github.com/docker/for-linux/issues/264#issuecomment-385698947

See:

• https://docs.docker.com/docker-for-mac/networking/#httphttps-proxy-support
• https://stackoverflow.com/questions/28056522/access-host-database-from-a-docker-container

Running commands with Open3#popen3 from, for example, a Rails application will give the command access to all ENV variables, including potential secrets.

Before running commands in you need to clean the ENV variables:

# Clean the ENV
def with_env(env)
  backup = ENV.to_hash
  ENV.replace(env)
  yield
ensure
  ENV.replace(backup)
end
# Set PWD, ENV, and options
pwd = '/Alset/YledoM/'
env = {
  'oLLEH' => 'DLROw',
  'RACK_ENV' => 'test',
  'PATH' => ENV['PATH']
}
options = {
  chdir: pwd
}
# The command to run
cmd = "bundle exec ruby script.rb"
# Prints a 'dirty' env
ENV.sort.to_h.each do |key, val|
  puts "#{key} => #{val}"
end
# Run the command with the given ENV
with_env(env) do
  Open3.popen3(env, cmd, options) do |stdin, stdout, stderr, wait_thr|
    # YXES
  end
  # Prints a clean env
  ENV.sort.to_h.each do |key, val|
    puts "#{key} => #{val}"
  end
end

POC for scripting Cucumber:

require 'cucumber'
opts = {
  strict: Cucumber::Core::Test::Result::StrictConfiguration.new,
  require: ["features/support", "features/step_definitions"],
  dry_run: false,
  formats: [['json', {}, outs]], # This is not working
  excludes: [],
  tag_expressions: [],
  tag_limits: {},
  name_regexps: [],
  env_vars: {},
  diff_enabled: true,
  snippets: true,
  source: true,
  duration: true,
  retry: 0,
  default_profile: "default"
}
configuration = Cucumber::Configuration.new(opts)
runtime = Cucumber::Runtime.new(configuration)
Cucumber::Cli::Main.new([]).execute!(runtime)