kubectl snippets

How to debug CrashLoopBackOff

Tagged crashloopbackoff, kubectl, kubernetes  Languages bash, yaml

One day, you see CrashLoopBackOff in the kubectl output:

$ kubectl get pod
NAME                               READY   STATUS             RESTARTS   AGE
app-548c9ddc46-z2fng               0/1     CrashLoopBackOff   79         6h26m

You already know that executing bash in the container is not possible because the container has crashed:

$ kubectl exec -ti app-548c9ddc46-z2fng bash
error: unable to upgrade connection: container not found ("app")

Option 1: Analyze the container logs

You can view the container’s logs with kubectl logs:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 2: Modify Dockerfile’s CMD

Modifying the Dockerfile’s ‘CMD’ is not needed, as it can be done indirectly through the Pod’s YAML configuration file:

apiVersion: v1
kind: Pod
  name: app
    - name: app
      image: company/app
      # DEBUG env variables
      command: [ "/bin/sh", "-c", "env" ]

The modified command will print the environment variables to the logs:

command: [ "/bin/sh", "-c", "env" ]

To view the output run:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 3: Start a shell and sleep (CMD)

Sleeping usually helps, and this can be done by modifying the container’s command:

apiVersion: extensions/v1beta1
kind: Deployment
  name: xxx-service
  replicas: 1
      - image: yyy/xxx:1.0.0
        name: xxx-service
          - "sh"
          - "-c"
          - "sleep 10000"

The container will start and run “sleep 10000” in a shell, giving you exactly 10000 seconds to debug the issue by connecting to the “sleeping” container:

$ kubectl exec -ti app-548c9ddc46-z2fng bash

Bash aliases / functions for kubectl

Tagged aliases, kubectl, k8s  Languages bash

Put the following functions in, for example, ~/.bash_aliases:

# Open a bash shell in the pod
function kshell() { kubectl exec -it "$1" -- /bin/bash ;}
# List pods
function kpods() { kubectl get po -n default -o wide ;}
function klogs() { kubectl logs -f "$1" ;}
function ktail() { kubectl logs --tail=100 -f "$1" ;}
# Restarts a pod
function krestart() { kubectl rollout restart "deployment/$1" ;}
# Stops a pod by scaling it to zero
function kstop() { kubectl scale --replicas=0 "deployment/$1" ;}
function kdesc() { kubectl describe pod $1 ;}
function ksecret() { kubectl get secret $1 -o jsonpath="{.data.$2}" | base64 --decode ;}
function ksecrets() { kubectl get secret/$1 -o yaml ;}

Your RSI symptoms and cognitive dissonance will disappear in about 6 months.

How to solve "CreateContainerConfigError" in Kubernetes

Tagged createcontainerconfigerror, k8s, kubectl  Languages bash

To solve CreateContainerConfigError try this command:

$ kubectl describe pod xxx-86cbd7555b-2zrs2
  Warning  Failed     75s (x8 over 2m50s)  kubelet, k8s-node1  Error: couldn't find key XXX in ConfigMap default/xxx