At Fullstory we make wide use of Kubernetes, the open-source container orchestration platform. Kubernetes is the base of almost all of our internal infrastructure, and powers multiple clusters across our environments. As a company, we have developed quite a list of repeated, complex actions that are executed against a variety of clusters. These are typically read-only actions, such as extracting or calculating a piece of information from a given cluster. We wanted a way to make these new commands easy-to-use by the rest of the team.
These complex, oft-repeated tasks are a perfect use case for writing our own custom `kubectl` plugins. Custom plugins let us neatly wrap up repeated actions against Kubernetes clusters in a discoverable way for the rest of our team.
As a quick refresher, a kubectl plugin is simply an executable on your `$PATH` which is prefixed with “kubectl-”; an executable named “kubectl-foo” is accessible by running `kubectl foo`.
As an example, in the past we had been fighting with a noticeable increase in `OOMKilled` errors across a couple of our services, which occur when a container attempts to use more memory than has been allocated to it. In order to dive deeper, we needed to know which particular pod was impacted, however this information was not available through our alerting pipeline at the time. It was common for our engineers to pass around large `jsonpath` schema selectors to use with `kubectl get`. To make this easier for everyone we created a quick script to identify and sort pods with recent restarts. The basic logic of our `kubectl-lastrestarts` plugin is:
Make a Kubernetes API call to list all pods
Extract the time of the last exit
Print out the last container exit time, along with the reason.
After adding this basic plugin to our path, we are now able to get the results we need with the command `kubectl lastrestarts`.
Side note: If you are more familiar with the workings of Kubernetes, you may notice that this approach does not allow us to track events historically since some pods may be missing if they have been removed since their exit. This is why a proper monitoring stack is a necessity for accurate tracking of these events. For the use cases that our team was encountering, potentially missing pods are something that we are able to overlook.
Distributing Kubectl Plugins
There are multiple ways to distribute kubectl plugins. In “the wild”, there is an impressive list of existing plugins available from the Krew plugin repository. If you want to explore and use public plugins, or publish your own plugins to the world, this is likely the path you want to take.
At Fullstory, we distribute our kubectl plugins as part of our existing monorepo workflow. We are already using direnv to control the environment within our development environments, and our existing build routine places all of our build binaries onto the $PATH for the project. This ensures that any engineer, after building the workspace, has our entire fleet of kubectl plugins available to them.
Do-s and Don’t-s of Plugins
Kubectl plugins are very handy for quickly addressing a specific problem. Much of the convenience comes from the familiarity of the existing tooling, which in this case is kubectl. We could get the same functionality by invoking a script directly, but that adds more cognitive load and complexity to the user of these tools since they need to remember which tools exist, how each one behaves, and where they live. You lose the kubectl functionality that you are already familiar with. This means that your plugins should behave as closely to a “native” command as possible.
Tips:
Do Accept any common `kubectl` arguments that your team uses. `-n <namespace`, `-A` (all namespaces), and `--context` should be accepted when applicable.
Do use a kubernetes client, rather than sub-shell-ing kubectl commands. This makes your plugin less environment-dependent.
If you are shelling out to kubectl subprocess: explicitly set `--context` if you are making multiple underlying `kubectl` command calls. You cannot assume that the global `current-context` will not change during a longer-running plugin command.
Do not change the current global `kubectl` context within your script
Do not use existing `kubectl` arguments for non-standard purposes
The Power of kubectl Plugins
Custom kubectl plugins are a great way to consolidate common Kubernetes management actions in a way that are easily distributed across teams. Our kubectl-lastrestarts plugin has simplified Kubernetes cluster debugging and management and increased the speed that we can diagnose problems with our services.
