Observability for Cloud Native
Integrating Honeycomb into your Kubernetes cluster with ksonnet
Although JSON/YAML Kubernetes manifests are straightforward to read and write, they are not always the best way to manage applications on your cluster. If you have a complex, production system and want to modify its deployment with existing approaches, you may experience significant operational costs.
In that case, what do you do if you want to add a feature like observability to your Kubernetes cluster? Even if you have a solution that (1) provides insight into the intrinsically dynamic workloads of a cloud native platform, it also needs to be (2) easy to embed and (3) easily extensible.
Recently we teamed up with the folks at Honeycomb, who had prior domain experience from Facebook, to address these points. Fortunately for us, they have an existing observability agent that handles all of (1). What we bring to the table is ksonnet, an open-source Jsonnet library and a powerful, composable approach to writing Kubernetes manifests. Our resulting collaboration, a Honeycomb library of ksonnet mixins, accomplishes all the aforementioned goals.
With ksonnet’s JSON-compatibility and powerful composability, you can configure and add Honeycomb to your existing app deployment, with only a few lines of code:
The Honeycomb mixin library, which uses the containerized Honeycomb agent, is available in the ksonnet/mixins repository.
For specific examples of how this library might change your approach to observability, read Honeycomb’s accompanying blog post.
Design Principles
ksonnet makes it easy to integrate Honeycomb into your cluster because it was explicitly created with two design principles in mind:
- Abstraction: ksonnet’s source code is imperative — variables, functions, and other Jsonnet syntax enable you to “config your config” and separate out your logic. (At the same time, because ksonnet compiles to a single Kubernetes-compatible JSON file, you still get the readability advantage of declarative programming.)
- Composability: Unlike existing tools, which mainly offer simple templating, ksonnet allows you to define core pieces of functionality in a set of building-block primitives (e.g., as defined in Honeycomb’s mixin library). Simple, algebraic operations make it easy to reuse primitives to create entirely new manifests, or to append functionality to an existing object.
This results in various benefits when you are using the Honeycomb mixins to build your own custom observability solution.
Compatibility and Concise Syntax
ksonnet has an upfront caveat: it can be slower to implement if you don’t already have a sense of what you’re building and its constituent patterns. In such cases, it may be easier to bash out a raw YAML config.
That being said, because ksonnet fundamentally wraps the same API as native Kubernetes manifests — as opposed to defining a brand new interface — the core mixins follow a familiar schema. The Visual Studio extension provides an autocomplete function that makes writing ksonnet more straightforward. Its generated YAML previews also allow you to reference your result throughout the prototyping process.
ksonnet’s syntax also allows you to avoid repetition. For example, the ksonnet code snippet introduced earlier in this post can generate a complete JSON manifest, despite its compactness. In particular, if you wanted to enable log file parsing by the Honeycomb agent, normally you would need to define a long series of volumes and volume mounts in your config file. These definitions are redundant (and typo-risky) to handle manually, especially if your deployment defines multiple containers.
However, with ksonnet, you can simply add a core Honeycomb mixin (honeycomb.app.deploymentBuilder.configureForPodLogs) to your deployment definition instead. It easily automates away this entire process, in a way that is still very much under your control.
Separation of Concerns
In YAML or JSON, there is no real ability to compose logic or tuck away redundant parts. The closest you can get to “organization” is grouping file components in a certain order. This means that there is no real separation between “external” values — variables unique to your deployment that will need to be swapped out accordingly — and “internal” values — reusable scaffolding that will rarely change.
ksonnet, in contrast, allows you to define your own layers of abstraction. For example, one of the sample Honeycomb manifests — the code snippet from earlier in this post — defines the following separation of concerns:
- A “meta-config” object: This is essentially a “config for the config”, and allows you to set values that are specific to your Honeycomb implementation (e.g. the name of a ConfigMap).
- A composition of core mixins: Mixins can take in the “meta-config” as an argument, and be merged with other mixins using the + operator. They span the range from app-level (e.g. DeploymentBuilder) to internal parts (e.g. rbac) . By selectively combining any number of these, you can define custom Kubernetes object(s).
This separation of concerns makes it easy to tweak some aspects of your Honeycomb app while effectively treating others as a black box. Consequently you can set up basic observability almost instantaneously. As your requirements change, you can easily learn more about previously abstracted parts.
Extensions
The upfront cost of ksonnet’s learning curve is quickly offset by the long-term advantages of extensibility.
For instance, it is generally unrealistic to expect any team to build tools that solve the entire power set of everyone’s unique observability needs — at least by themselves. There are just too many possibilities that need to be accounted and configured for. However, Honeycomb solves this problem by defining a library of core mixins — and essentially providing the “keys” to the user. Because of ksonnet’s powerful composability, you can easily mix-and-match the core mixins in Honeycomb’s library to build your own custom solution. It blurs the distinction between implementer and user.
Whether your use case requires either (1) a recombination of existing mixin features or (2) an entirely new feature, ksonnet provides a relatively principled way to do both, without having to copy and paste existing manifests. (1) is as simple as invoking the necessary components and merging them with the + operator. The only additional step required for (2) is to write your own mixin, to combine with core Honeycomb mixins. The + operator does not discriminate between DIY mixins and official mixins.
Because customization now has low operational costs, you can easily adapt your Honeycomb installation to address your observability needs on the fly. You may, for example, find that you prefer to have the Honeycomb agent running as a sidecar container in a Deployment, rather than as a DaemonSet on all your nodes. Mixins allow your application-level components to stay decoupled from any particular implementation. You are able to leverage Honeycomb’s features and maintain agency over your own observability.
Moving Forward
We hope that this post has given you a taste of how effective ksonnet mixins can be — for observability in particular, as well as other potential domains.
- If you are interested in using the Honeycomb monitoring agent, please reach out to their team! They are amazing folks to work with. :)
- If you have not used ksonnet before, we encourage you to try it out! You can visit the Github page, check out our interactive ksonnet playground, or follow the tutorial. Ksonnet is in active development, so stay tuned for new features and abstractions as they are released.
- If you have your own ideas for ksonnet libraries that you think can be widely leveraged, please submit your code as a pull request to the mixins repo. Your experiences can help steer feature development and codify a set of ksonnet best practices. Active developer communities are incredibly creative, so we’re excited to see where you take this!
