Kubernetes tooling for TechOps and Support

April 15, 2021 | By Tom Halligan

Backend of a kubernetes tool

Introduction:

One of the challenges organizations face while adopting Kubernetes is providing Operations/Support personnel with the tools and training they need to support K8s deployments. Kubernetes adoption is often driven by development or engineering teams and those teams tend to use tooling that maps to their needs but may not map to breakfix support functions.

Support teams need to ramp up quickly on Kubernetes while continuing to support existing environments, therefore it is important that teams who provide tier 1/2 support of K8s have proper foundational training in Kubernetes. Management tools are of little use if the basics of K8s are not understood. Support teams need to focus on K8s architecture and basic principles. Oteemo has a great training program to introduce teams to Kuberneteshttps://oteemo.com/enterprise-enablement/

I am touching on two tools, one that assists with K8s self-training and a second tool that is an excellent fit for break fix support of K8s.

Self Paced Training with K3d (https://github.com/rancher/k3d):

To assist in self learning there are single node k8s implementations (minikube) and cloud provided solutions that can be used with minimal expense (EKS, GKE etc) but the best, in my opinion, from light weight, quick deploy, and emulation of function standpoint is K3d a Dockerized Kubernetes cluster based on the K3s distribution maintained by Rancher.

     

      • K3d runs on MACs and Linux OS for full list of install instructions check out the Github site.

      • Can emulate a multi-node cluster on your local system.

      • K3d comes with traefik installed which allows you to work with ingresses, see –publish flag.

      • Deployments are very fast, 15 seconds or so.

      • State is maintained between restarts.

    Below is a quick start guide (Docker needs to already be installed on your system)

    On a MAC run brew install k3d to install the tool on Linux run curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash

    Once install is complete you can bring up your 3 node cluster

    k3d create –workers 2  –name somename

    Then adjust your kubeconfig

    export KUBECONFIG=“$(k3d get-kubeconfig –name=‘somename’)”

    Run kubectl get nodes and you will see your cluster.

    NAME                    STATUS ROLES AGE VERSION
    k3d-somename-worker-0   Ready 13s v1.16.2-k3s.1
    k3d-somename-server     Ready master 14s v1.16.2-k3s.1
    k3d-somename-worker-1   Ready 13s v1.16.2-k3s.1

    You can now practice kubectl commands, simple deployments, tainting and labels for scheduling etc.

    When you are done you can run the following to cleanup (But if you are following along let’s leave this cluster up to use with next tool)

    k3d delete –name somename

    Operations/Support with K9s (https://k9ss.io/):

    There are a number of management tools for Kubernetes clusters, but for my money the tool that most closely maps to day to day operations tasks is K9s.

       

        • The tool is a self contained Go binary and can run on Mac, Windows and Linux.

        • Installation is  simple and easy to ingrate into a build workflow, or quickly install on a system, with a minimal resource footprint.

        • You can switch between different contexts (K8s clusters) within the tool, and isolateswitch namespaces.

        • You can describe, edit, scale and kill resources through the UI (Not the preferred way to make changes but sometimes a necessity for support purposes)

        • The GUI is curses based and lightweight allowing the operator to move quickly between resource types (Use CTRL-A to get the full list)

      So let’s do a quick example using the k3d cluster we setup above. First we will create a couple of deployments one good one bad

      Run:
      kubectl run nginxgood –image=nginx –replicas=2
      kubectl run nginxbad –image=nginy –replicas=2

      Then lets install k9s and start it up

      On a MAC you can run brew install derailed/k9s/k9s or you can download binary for Linux, Windows or MAC https://github.com/derailed/k9s/releases and put it in your path

      Then run k9s

      The UI will default to the pod view and we can see the pods from our two deployments, one set of pods is failing. You can select any of the pods and describe pod d, get logs l, shell into container running on pod s (You can select a pod hit enter and get a list of all containers in pod and shell into any of them), and even use CTRL-k to kill a pod.

      k9s main screen

      For our case pod describe d shows us what is wrong, a typo on the image name.

      k9s describe pod

      To switch between resource views you hit : which brings up command interpreter and then put in the short name for the resource (CTRL-a gives full list), in our case we are going to deployments so dp

      k9s switch

      We can then select the owner deployment and hit e to edit the deployment manifest (Will pop you into vi) we can then correct the typo and :wq!

      k9s deployment

      deply edit

      Then we can switch back to pod view : po and see the deployment initiate new pods and terminate the old ones.

      k9s good pods

      Conclusion:

      The key to supporting any complex system is understanding, and that is where training/tooling is critical. Some implementations of applications in K8s can be wholly contained in CI/CD process that negates the need for interventionist support (When there is an issue you fix in code and redeploy). A significant number of organizations have not achieved that level of sophistication and K8s deployments will need support from techops resources, lest the Dev teams get overwhelmed with interruptive support work.

      The above tools can assist support teams to rapidly learn the basics of K8s management and troubleshooting and quickly address level 1/2 issues that may occur.

      Addendum:

      Both k3d and k9s are great tools for development purposes also, with k9s especially have many great features (Custom kubectl plugins), they are not Operator specific, but great tools for a DevOPs organization to have on hand.

      5 Comments

      1. Luis A Angarita

        Tom this was a great blog! It brought me up to speed with k3d and k9s quickly and with ease.
        Thanks for putting this thing together.

        • Tom Halligan

          Glad it was helpful

      2. Luca Ravazzolo

        Great post! Thank you guys!
        Also, I was looking for the article on Mozilla sops… it has vanished 🙁 Please let me know. thanks

      3. LG

        How could you not talk about Lens ?! after using k9S since the beginning, I’m become so much more efficient with Lens. You have the GUI, the alerting + the console if needed

        • Tom Halligan

          This post was directed towards support and Ops functions and I did not do a good job calling out why k9s is better suited to that use case. I use Lens when locally working on my workstation but the cli friendly nature of k9s, plus standalone binary, makes it perfect when running from jumphost or gateway system where all you might get is a shell.

          Also when in troubleshooting mode on my workstation I often find myself switching back to k9s, which may be because I am more cli inclined and can move a bit faster in that space, rather any advantage one tool has over another.

          Thanks for the comment and I will see if I can update the post with a bit more detail on why I think K9s is more suited to a troubleshooting role when you don’t necessarily have any say in the environment you are working in.

      oteemo

      Who We Are & What We Do

      As passionate technologists, we love to push the envelope. We act as strategists, practitioners and coaches to enable enterprises to adopt modern technology and accelerate innovation.

      We help customers win by meeting their business objectives efficiently and effectively.

      icon         icon        icon

      Newsletter Signup:

      Join tens of thousands of your peers and sign-up for our best technology content curated by our experts. We never share or sell your email address!