kubectl snippets

How to debug CrashLoopBackOff

Tagged crashloopbackoff, kubectl, kubernetes  Languages bash, yaml

One day, you see CrashLoopBackOff in the kubectl output:

$ kubectl get pod
NAME                               READY   STATUS             RESTARTS   AGE
app-548c9ddc46-z2fng               0/1     CrashLoopBackOff   79         6h26m

You already know that executing bash in the container is not possible because the container has crashed:

$ kubectl exec -ti app-548c9ddc46-z2fng bash
error: unable to upgrade connection: container not found ("app")

Option 1: Analyze the container logs

You can view the container’s logs with kubectl logs:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 2: Modify Dockerfile’s CMD

Modifying the Dockerfile’s ‘CMD’ is not needed, as it can be done indirectly through the Pod’s YAML configuration file:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: company/app
      # DEBUG env variables
      command: [ "/bin/sh", "-c", "env" ]

The modified command will print the environment variables to the logs:

command: [ "/bin/sh", "-c", "env" ]

To view the output run:

$ kubectl logs -f app-548c9ddc46-z2fng

Option 3: Start a shell and sleep (CMD)

Sleeping usually helps, and this can be done by modifying the container’s command:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: xxx-service
spec:
  replicas: 1
  template:
    ...
    spec:
      containers:
      - image: yyy/xxx:1.0.0
        name: xxx-service
        ...
        command:
          - "sh"
          - "-c"
          - "sleep 10000"

The container will start and run “sleep 10000” in a shell, giving you exactly 10000 seconds to debug the issue by connecting to the “sleeping” container:

$ kubectl exec -ti app-548c9ddc46-z2fng bash

Bash aliases / functions for kubectl

Tagged aliases, kubectl, k8s  Languages bash

Put the following functions in, for example, ~/.bash_aliases:

function kshell() { kubectl exec -it "$1" -- /bin/bash ;}
function kpods() { kubectl get po -n default -o wide ;}
function klogs() { kubectl logs --tail=100 -f "$1" ;}
function krestart() { kubectl rollout restart "deploy/$1" ;}
function kdesc() { kubectl describe pod $1;}
function ksecret() { kubectl get secret $1 -o jsonpath="{.data.$2}" | base64 --decode ;}

Your RSI and symptoms of cognitive dissonance will disappear in about 6 months.

How to solve "CreateContainerConfigError" in Kubernetes

Tagged createcontainerconfigerror, k8s, kubectl  Languages bash

To solve CreateContainerConfigError you can try this command:

$ kubectl describe pod xxx-86cbd7555b-2zrs2
...
...
  Warning  Failed     75s (x8 over 2m50s)  kubelet, k8s-node1  Error: couldn't find key XXX in ConfigMap default/xxx
...