Introduction:

One of the challenges organizations face while adopting Kubernetes is providing Operations/Support personnel with the tools and training they need to support K8s deployments. Kubernetes adoption is often driven by development or engineering teams and those teams tend to use tooling that maps to their needs but may not map to breakfix support functions.

Support teams need to ramp up quickly on Kubernetes while continuing to support existing environments, therefore it is important that teams who provide tier 1/2 support of K8s have proper foundational training in Kubernetes. Management tools are of little use if the basics of K8s are not understood. Support teams need to focus on K8s architecture and basic principles. Oteemo has a great training program to introduce teams to Kubernetes- https://oteemo.com/training-kubernetes/

I am touching on two tools, one that assists with K8s self-training and a second tool that is an excellent fit for break fix support of K8s.

Self Paced Training with K3d (https://github.com/rancher/k3d):

To assist in self learning there are single node k8s implementations (minikube) and cloud provided solutions that can be used with minimal expense (EKS, GKE etc) but the best, in my opinion, from light weight, quick deploy, and emulation of function standpoint is K3d a Dockerized Kubernetes cluster based on the K3s distribution maintained by Rancher.

• K3d runs on MACs and Linux OS for full list of install instructions check out the Github site.
• Can emulate a multi-node cluster on your local system.
• K3d comes with traefik installed which allows you to work with ingresses, see –publish flag.
• Deployments are very fast, 15 seconds or so.
• State is maintained between restarts.

Below is a quick start guide (Docker needs to already be installed on your system)

On a MAC run brew install k3d to install the tool on Linux run curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash

Once install is complete you can bring up your 3 node cluster

k3d create –workers 2  –name somename

Then adjust your kubeconfig

export KUBECONFIG=“$(k3d get-kubeconfig –name=‘somename’)”

Run kubectl get nodes and you will see your cluster.

NAME                    STATUS ROLES AGE VERSION
k3d-somename-worker-0   Ready 13s v1.16.2-k3s.1
k3d-somename-server     Ready master 14s v1.16.2-k3s.1
k3d-somename-worker-1   Ready 13s v1.16.2-k3s.1

You can now practice kubectl commands, simple deployments, tainting and labels for scheduling etc.

When you are done you can run the following to cleanup (But if you are following along let’s leave this cluster up to use with next tool)

k3d delete –name somename

Operations/Support with K9s (https://k9ss.io/):

There are a number of management tools for Kubernetes clusters, but for my money the tool that most closely maps to day to day operations tasks is K9s.

• The tool is a self contained Go binary and can run on Mac, Windows and Linux.
• Installation is  simple and easy to ingrate into a build workflow, or quickly install on a system, with a minimal resource footprint.
• You can switch between different contexts (K8s clusters) within the tool, and isolateswitch namespaces.
• You can describe, edit, scale and kill resources through the UI (Not the preferred way to make changes but sometimes a necessity for support purposes)
• The GUI is curses based and lightweight allowing the operator to move quickly between resource types (Use CTRL-A to get the full list)

So let’s do a quick example using the k3d cluster we setup above. First we will create a couple of deployments one good one bad

Run:
kubectl run nginxgood –image=nginx –replicas=2
kubectl run nginxbad –image=nginy –replicas=2

Then lets install k9s and start it up

On a MAC you can run brew install derailed/k9s/k9s or you can download binary for Linux, Windows or MAC https://github.com/derailed/k9s/releases and put it in your path

Then run k9s

The UI will default to the pod view and we can see the pods from our two deployments, one set of pods is failing. You can select any of the pods and describe pod d, get logs l, shell into container running on pod s (You can select a pod hit enter and get a list of all containers in pod and shell into any of them), and even use CTRL-k to kill a pod.

For our case pod describe d shows us what is wrong, a typo on the image name.

To switch between resource views you hit : which brings up command interpreter and then put in the short name for the resource (CTRL-a gives full list), in our case we are going to deployments so dp

We can then select the owner deployment and hit e to edit the deployment manifest (Will pop you into vi) we can then correct the typo and :wq!

Then we can switch back to pod view : po and see the deployment initiate new pods and terminate the old ones.

Conclusion:

The key to supporting any complex system is understanding, and that is where training/tooling is critical. Some implementations of applications in K8s can be wholly contained in CI/CD process that negates the need for interventionist support (When there is an issue you fix in code and redeploy). A significant number of organizations have not achieved that level of sophistication and K8s deployments will need support from techops resources, lest the Dev teams get overwhelmed with interruptive support work.

The above tools can assist support teams to rapidly learn the basics of K8s management and troubleshooting and quickly address level 1/2 issues that may occur.

Addendum:

Both k3d and k9s are great tools for development purposes also, with k9s especially have many great features (Custom kubectl plugins), they are not Operator specific, but great tools for a DevOPs organization to have on hand.