Setup lightweight Kubernetes with K3s

K3s is a lightweight Kubernetes distribution, which looks like easy to install for devops, but not for dummies. This article gives answers for questions: How to solve install error? How to configure ingress? How to use dashboards? How to force Docker? How to use already installed kubectl?

K3s uses a built-in containerd as the container runtime, by default. This default setup has more advantages:

• Docker is not needed.
• If Docker is installed, the K3s containers and Docker containers are running in separated environment and cannot influence each other

Below description forces to use Docker to give opportunity for Docker experts to see what K3s makes under the hood. Of course, below example works without Docker, too (noticed at right place).

The default K3s ingress controller is Traefik, it’s not changed.

Used versions:

• K3s: v1.19.5+k3s2, see built-in versions at https://github.com/k3s-io/k3s/releases/tag/v1.19.5%2Bk3s2
• Kubernetes Dashboard: v2.1.0
• Nginx docker image: latest

Tested on:

• Ubuntu 16.04, bare metal (Intel Celeron N2830, 4 GB RAM)
• Ubuntu 18.04, VM (12 GB RAM), running in Windows 10 Hyper-V

Warning: Below example is not secure and must be hardened before using it in production.

If you are interested in a more complex K8s deployment, you can read my next article Setup On-premise Kubernetes with Kubeadm, MetalLB, Traefik and Vagrant . These two K8s clusters can run same time on a Linux bare metal machine, so it’s good opportunity to compare a lightweight K8s development environment and a VM-based multi-node on-premise K8s deployment.

Prerequisites

If you want to force Docker, Docker CE must be installed, for example:

• https://docs.docker.com/engine/install/ubuntu/
• curl https://releases.rancher.com/install-docker/19.03.sh | sh

The TCP port 80 must be free (for ingress).

For homeworkers: if you are using VPN, a few K3s subnets (10.42.0.0/16 and 10.43.0.0/16) shouldn’t be in the VPN-routed address space. See more details at https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking .

Install K3s

If you don’t want to force Docker, you should skip --docker option below.

If you don’t have installed kubectl, then (makes link to built-in K3s kubectl, by default):

curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 644

else:

curl -sfL https://get.k3s.io | INSTALL_K3S_SYMLINK=skip sh -s - --docker --write-kubeconfig-mode 644
# Execute more steps: https://rancher.com/docs/k3s/latest/en/cluster-access/ , for example:
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Note: the --write-kubeconfig-mode 644 option is needed to avoid kubectl permission denied error later on.

Configure kubectl autocomplete

If kubectl autocomplete is not set yet:

echo 'source <(kubectl completion bash)' >>~/.bashrc
source <(kubectl completion bash)

Check K3s install

k3s check-config
kubectl cluster-info
kubectl get nodes -o wide
# Wait until all pods and deployments aren't Running or Completed,
# see READY and STATUS columns
kubectl get all -A -o wide
kubectl get endpoints -A
sudo k3s crictl ps -a
kubectl top pod --containers -A
# Below command works only if Docker has already been installed
docker ps

Dashboards

Dashboards are not needed, but easy for getting information about Kubernetes resources.

Dashboards can be accessed by several ways. Below chapters describe 2 ways, one of them is enough:

• kubectl proxy, trough http://localhost:8001/api/v1/namespaces/<NAMESPACE>/services/<HTTP_PROTO>:<SERVICE>:<PORT>/proxy/ pattern
• Trough ingress controller

If ingress controller is configured, dashboards can be accessed trough a dedicated hostname oam.internal. Open hosts file:

sudo nano /etc/hosts

Add below line and save+exit:

127.0.1.2       oam.internal

Install Traefik Dashboard

Optional.

Traefik Dashboard is not enabled by default. It can be enabled in Helm file of Traefik.

Open Helm file:

sudo nano /var/lib/rancher/k3s/server/manifests/traefik.yaml

Append dashboard lines to spec.valuesContent, like:

(...)
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
  valuesContent: |-
    dashboard:
      enabled: true
      domain:  "oam.internal"
    rbac:
      enabled: true
(...)

After save+exit, wait for an existing endpoint:

watch kubectl get endpoints traefik-dashboard -n kube-system

Note: after reboot the Helm file and the ingress config are recovered to the original content.

Access Traefik Dashboard trough kubectl proxy

Start kubectl proxy in a new terminal:

kubectl proxy

Open http://localhost:8001/api/v1/namespaces/kube-system/services/http:traefik-dashboard:80/proxy/dashboard/#/ in a browser.

Access Traefik Dashboard trough ingress

Ingress has already been configured by changing Helm file. The Helm chart does not support to set path in ingress config, so it’s bound to the root.

Open http://oam.internal/dashboard/ in a browser.

Install Kubernetes Dashboard

Optional.

Deploying the Kubernetes Dashboard:

GITHUB_URL=https://github.com/kubernetes/dashboard/releases
VERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/alternative.yaml

Dashboard RBAC configuration:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

Wait for an existing endpoint:

watch kubectl get endpoints kubernetes-dashboard -n kubernetes-dashboard

Obtain the Bearer Token (output will be used for login):

kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token

Access Kubernetes Dashboard trough kubectl proxy

Start kubectl proxy in a new terminal:

kubectl proxy

Open http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in a browser.
Use the bearer token printed out at earlier step.

Access Kubernetes Dashboard trough ingress

Configure ingress controller:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/ingress.class: "traefik"
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
spec:
  rules:
  - host: oam.internal
    http:
      paths:
      - path: /kubernetes
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 80
EOF

Open https://oam.internal/kubernetes/ in a browser.
Use the bearer token printed out at an earlier step.

Setup a simple web server

Based on: https://opensource.com/article/20/3/kubernetes-traefik , but shorter.

Deploy and expose web server

Below 2 commands supplements the deprecated/removed deployment generator in kubectl run :

kubectl create deployment --image nginx:latest my-nginx
kubectl expose deployment my-nginx --port=80

Note: all generators are deprecated/removed in kubectl run command (depending on K8s version), see more details at https://kubernetes.io/docs/reference/kubectl/conventions/#kubectl-run and https://github.com/kubernetes/kubernetes/pull/87077

Check web server

kubectl get all -o wide
# Wait until deployment.apps/my-nginx READY is not 1/1
# Use pod NAME from above output in below command
kubectl get pod/my-nginx-b7d7bc74d-7mhdf -o yaml
kubectl get deployment.apps/my-nginx -o yaml
kubectl get service/my-nginx -o wide
# Use CLUSTER-IP and PORT from above output in below command
curl 10.43.234.100:80

Checking service DNS:

kubectl run cluster-tester -it --rm --restart=Never --image=busybox:1.28
#Or: kubectl run cluster-tester -it --rm --restart=Never --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
nslookup kubernetes.default
nslookup my-nginx.default.svc.cluster.local
wget -qSO- my-nginx.default.svc.cluster.local
#For dnsutils: wget -qO- my-nginx.default.svc.cluster.local
exit

Custom configuration

Optional.

If you would like to deploy and expose web server by kubeclt apply, as it written in the base documentation, you can start it with output of below harmless commands:

kubectl create deployment --image nginx:latest my-nginx --dry-run=client -o yaml
kubectl expose deployment my-nginx --port=80 --dry-run=client -o yaml

The outputs can be compared to kubectl run output (without deployment generator):

kubectl run my-nginx --image=nginx --port=80 --expose=true --dry-run=client -o yaml

Publish web server

Below commands expose a HTTP route from outside to Kubernetes my-nginx service:

cat <<EOT >> /tmp/mysite.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mysite-nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
spec:
  rules:
  - http:
      paths:
      - path: /my-nginx
        pathType: Prefix
        backend:
          service:
            name: my-nginx
            port:
              number: 80
EOT
kubectl apply -f /tmp/mysite.yaml
curl $(hostname --fqdn)/my-nginx

Open hostname --fqdn output + /my-nginx URL in a browser.

Scaling web server

Optional.

Scaling simple web server to 3 instances:

kubectl scale --replicas=3 deployment/my-nginx
kubectl get deployment/my-nginx -o wide
# Wait until deployment.apps/my-nginx READY is not 3/3

Cleanup web server

Optional.

All resources are created by kubectl apply -f can be deleted by below command:

kubectl delete -f /tmp/mysite.yaml

The rest resources can be deleted by below commands (if not deleted by kubectl delete -f earlier):

kubectl delete ingress/mysite-nginx-ingress
kubectl delete service/my-nginx
kubectl delete deployment/my-nginx

Uninstall K3s

Optional.

/usr/local/bin/k3s-uninstall.sh
sudo rm -rf /var/lib/rancher/k3s/ /etc/rancher/k3s

References

• https://github.com/k3s-io/k3s
• https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking
• https://rancher.com/docs/k3s/latest/en/advanced/#using-docker-as-the-container-runtime
• https://rancher.com/docs/rancher/v2.x/en/installation/resources/installing-docker/
• https://rancher.com/docs/k3s/latest/en/installation/install-options/
• https://opensource.com/article/20/3/kubernetes-traefik
• https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/
• https://forums.rancher.com/t/k3s-traefik-dashboard-activation/17142/2
• https://blog.kubernauts.io/k3s-with-metallb-on-multipass-vms-ac2b37298589
• https://medium.com/@chamilad/load-balancing-and-reverse-proxying-for-kubernetes-services-f03dd0efe80
• https://www.disasterproject.com/kubernetes-with-external-dns/

More info about K3d:

• https://rancher.com/docs/k3s/latest/en/advanced/#running-k3d-k3s-in-docker-and-docker-compose
• https://github.com/k3s-io/k3s/blob/master/docker-compose.yml