Setup lightweight Kubernetes with K3s
--
K3s is a lightweight Kubernetes distribution, which looks like easy to install for devops, but not for dummies. This article gives answers for questions: How to solve install error? How to configure ingress? How to use dashboards? How to force Docker? How to use already installed kubectl?
K3s uses a built-in containerd as the container runtime, by default. This default setup has more advantages:
- Docker is not needed.
- If Docker is installed, the K3s containers and Docker containers are running in separated environment and cannot influence each other
Below description forces to use Docker to give opportunity for Docker experts to see what K3s makes under the hood. Of course, below example works without Docker, too (noticed at right place).
The default K3s ingress controller is Traefik, it’s not changed.
Used versions:
- K3s: v1.19.5+k3s2, see built-in versions at https://github.com/k3s-io/k3s/releases/tag/v1.19.5%2Bk3s2
- Kubernetes Dashboard: v2.1.0
- Nginx docker image: latest
Tested on:
- Ubuntu 16.04, bare metal (Intel Celeron N2830, 4 GB RAM)
- Ubuntu 18.04, VM (12 GB RAM), running in Windows 10 Hyper-V
Warning: Below example is not secure and must be hardened before using it in production.
If you are interested in a more complex K8s deployment, you can read my next article Setup On-premise Kubernetes with Kubeadm, MetalLB, Traefik and Vagrant . These two K8s clusters can run same time on a Linux bare metal machine, so it’s good opportunity to compare a lightweight K8s development environment and a VM-based multi-node on-premise K8s deployment.
Prerequisites
If you want to force Docker, Docker CE must be installed, for example:
- https://docs.docker.com/engine/install/ubuntu/
- curl https://releases.rancher.com/install-docker/19.03.sh | sh
The TCP port 80 must be free (for ingress).
For homeworkers: if you are using VPN, a few K3s subnets (10.42.0.0/16 and 10.43.0.0/16) shouldn’t be in the VPN-routed address space. See more details at https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking .
Install K3s
If you don’t want to force Docker, you should skip --docker
option below.
If you don’t have installed kubectl, then (makes link to built-in K3s kubectl, by default):
curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 644
else:
curl -sfL https://get.k3s.io | INSTALL_K3S_SYMLINK=skip sh -s - --docker --write-kubeconfig-mode 644# Execute more steps: https://rancher.com/docs/k3s/latest/en/cluster-access/ , for example:
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
Note: the --write-kubeconfig-mode 644
option is needed to avoid kubectl permission denied
error later on.
Configure kubectl autocomplete
If kubectl autocomplete is not set yet:
echo 'source <(kubectl completion bash)' >>~/.bashrcsource <(kubectl completion bash)
Check K3s install
k3s check-configkubectl cluster-infokubectl get nodes -o wide# Wait until all pods and deployments aren't Running or Completed,
# see READY and STATUS columns
kubectl get all -A -o widekubectl get endpoints -Asudo k3s crictl ps -akubectl top pod --containers -A# Below command works only if Docker has already been installed
docker ps
Dashboards
Dashboards are not needed, but easy for getting information about Kubernetes resources.
Dashboards can be accessed by several ways. Below chapters describe 2 ways, one of them is enough:
kubectl proxy
, trough http://localhost:8001/api/v1/namespaces/<NAMESPACE>/services/<HTTP_PROTO>:<SERVICE>:<PORT>/proxy/ pattern- Trough ingress controller
If ingress controller is configured, dashboards can be accessed trough a dedicated hostname oam.internal
. Open hosts file:
sudo nano /etc/hosts
Add below line and save+exit:
127.0.1.2 oam.internal
Install Traefik Dashboard
Optional.
Traefik Dashboard is not enabled by default. It can be enabled in Helm file of Traefik.
Open Helm file:
sudo nano /var/lib/rancher/k3s/server/manifests/traefik.yaml
Append dashboard lines to spec.valuesContent
, like:
(...)
spec:
chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
valuesContent: |-
dashboard:
enabled: true
domain: "oam.internal"
rbac:
enabled: true
(...)
After save+exit, wait for an existing endpoint:
watch kubectl get endpoints traefik-dashboard -n kube-system
Note: after reboot the Helm file and the ingress config are recovered to the original content.
Access Traefik Dashboard trough kubectl proxy
Start kubectl proxy in a new terminal:
kubectl proxy
Open http://localhost:8001/api/v1/namespaces/kube-system/services/http:traefik-dashboard:80/proxy/dashboard/#/ in a browser.
Access Traefik Dashboard trough ingress
Ingress has already been configured by changing Helm file. The Helm chart does not support to set path in ingress config, so it’s bound to the root.
Open http://oam.internal/dashboard/ in a browser.
Install Kubernetes Dashboard
Optional.
Deploying the Kubernetes Dashboard:
GITHUB_URL=https://github.com/kubernetes/dashboard/releasesVERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/alternative.yaml
Dashboard RBAC configuration:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
Wait for an existing endpoint:
watch kubectl get endpoints kubernetes-dashboard -n kubernetes-dashboard
Obtain the Bearer Token (output will be used for login):
kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
Access Kubernetes Dashboard trough kubectl proxy
Start kubectl proxy in a new terminal:
kubectl proxy
Open http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in a browser.
Use the bearer token printed out at earlier step.
Access Kubernetes Dashboard trough ingress
Configure ingress controller:
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kubernetes-dashboard-ingress
namespace: kubernetes-dashboard
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
spec:
rules:
- host: oam.internal
http:
paths:
- path: /kubernetes
pathType: Prefix
backend:
service:
name: kubernetes-dashboard
port:
number: 80
EOF
Open https://oam.internal/kubernetes/ in a browser.
Use the bearer token printed out at an earlier step.
Setup a simple web server
Based on: https://opensource.com/article/20/3/kubernetes-traefik , but shorter.
Deploy and expose web server
Below 2 commands supplements the deprecated/removed deployment generator in kubectl run
:
kubectl create deployment --image nginx:latest my-nginxkubectl expose deployment my-nginx --port=80
Note: all generators are deprecated/removed in
kubectl run
command (depending on K8s version), see more details at https://kubernetes.io/docs/reference/kubectl/conventions/#kubectl-run and https://github.com/kubernetes/kubernetes/pull/87077
Check web server
kubectl get all -o wide
# Wait until deployment.apps/my-nginx READY is not 1/1# Use pod NAME from above output in below command
kubectl get pod/my-nginx-b7d7bc74d-7mhdf -o yamlkubectl get deployment.apps/my-nginx -o yamlkubectl get service/my-nginx -o wide# Use CLUSTER-IP and PORT from above output in below command
curl 10.43.234.100:80
Checking service DNS:
kubectl run cluster-tester -it --rm --restart=Never --image=busybox:1.28
#Or: kubectl run cluster-tester -it --rm --restart=Never --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3nslookup kubernetes.defaultnslookup my-nginx.default.svc.cluster.localwget -qSO- my-nginx.default.svc.cluster.local
#For dnsutils: wget -qO- my-nginx.default.svc.cluster.localexit
Custom configuration
Optional.
If you would like to deploy and expose web server by kubeclt apply
, as it written in the base documentation, you can start it with output of below harmless commands:
kubectl create deployment --image nginx:latest my-nginx --dry-run=client -o yamlkubectl expose deployment my-nginx --port=80 --dry-run=client -o yaml
The outputs can be compared to kubectl run
output (without deployment generator):
kubectl run my-nginx --image=nginx --port=80 --expose=true --dry-run=client -o yaml
Publish web server
Below commands expose a HTTP route from outside to Kubernetes my-nginx service:
cat <<EOT >> /tmp/mysite.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mysite-nginx-ingress
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
spec:
rules:
- http:
paths:
- path: /my-nginx
pathType: Prefix
backend:
service:
name: my-nginx
port:
number: 80
EOTkubectl apply -f /tmp/mysite.yamlcurl $(hostname --fqdn)/my-nginx
Open hostname --fqdn
output + /my-nginx
URL in a browser.
Scaling web server
Optional.
Scaling simple web server to 3 instances:
kubectl scale --replicas=3 deployment/my-nginxkubectl get deployment/my-nginx -o wide
# Wait until deployment.apps/my-nginx READY is not 3/3
Cleanup web server
Optional.
All resources are created by kubectl apply -f
can be deleted by below command:
kubectl delete -f /tmp/mysite.yaml
The rest resources can be deleted by below commands (if not deleted by kubectl delete -f
earlier):
kubectl delete ingress/mysite-nginx-ingresskubectl delete service/my-nginxkubectl delete deployment/my-nginx
Uninstall K3s
Optional.
/usr/local/bin/k3s-uninstall.shsudo rm -rf /var/lib/rancher/k3s/ /etc/rancher/k3s
References
- https://github.com/k3s-io/k3s
- https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking
- https://rancher.com/docs/k3s/latest/en/advanced/#using-docker-as-the-container-runtime
- https://rancher.com/docs/rancher/v2.x/en/installation/resources/installing-docker/
- https://rancher.com/docs/k3s/latest/en/installation/install-options/
- https://opensource.com/article/20/3/kubernetes-traefik
- https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/
- https://forums.rancher.com/t/k3s-traefik-dashboard-activation/17142/2
- https://blog.kubernauts.io/k3s-with-metallb-on-multipass-vms-ac2b37298589
- https://medium.com/@chamilad/load-balancing-and-reverse-proxying-for-kubernetes-services-f03dd0efe80
- https://www.disasterproject.com/kubernetes-with-external-dns/
More info about K3d: