An Introduction to Extending Kubernetes with CustomResourceDefinitions

Kubernetes provides a rich set of both primitives (e.g. pods) and higher-level constructs (e.g. deployments) you can use to run your workloads. It also has a solid technical foundation, providing an API server backed by etcd for data storage.

You can accomplish many things with the out of the box components, but sometimes the built-in functionality isn’t enough. Maybe you want to run a backup when a new Backup resource is created, or you want to modify an egress proxy’s configuration to enable a pod to use it when a new PodEgress resource is created, or … Whatever the reason, this post describes the different ways to extend Kubernetes, and provides an introduction to the simplest way to extend it — CustomResourceDefinitions.

Photo by Igor Ovsyannykov on Unsplash

API Extension Mechanisms

There are three ways to add new API types to the Kubernetes API server:

1. Modify the Kubernetes source code
2. Create a custom API server and aggregate it into the cluster
3. Write CustomResourceDefinitions

Writing CustomResourceDefinitions (CRDs) is the simplest approach. You may have heard of the previous experimental API extension mechanism called ThirdPartyResources (TPRs); CRDs replace TPRs and allow you to define an API group and version and all of the permissible resources for the new API. Although it has more limitations than the others, it’s often the right solution. Here are some of the limitations:

• CRDs only allow and support a single API version per resource.
• CRDs only support basic ObjectMeta validation in Kubernetes version 1.7 and earlier. Custom validation via JSON Schema is available starting in version 1.8.
• They don’t support sub-resources, including /status for status-only updates. But see this proposal to add /status and /scale sub-resources.
• OwnerReference-based garbage collection for CRDs is available starting in version 1.8. It isn’t available in earlier versions.
• CRDs only support json over the wire; there’s no support for protobuf.

Creating a custom API server is more flexible, but it also involves a lot more work both in writing a significant amount of code and in managing persistent storage. With a custom API server, you can have the usual resources, sub-resources, custom validation, and other logic — basically anything you can do in the Kubernetes API server, you can do in a custom API server.

But you have to write a significant amount of code to make sure everything is wired together correctly. Fortunately, a proof-of-concept project helps make the process as simple as possible.

And you still have to figure out persistent storage for your custom API server. While you could use the cluster’s etcd instance, doing so could be a security risk: your API server would essentially have “root” and be able to access anything it wanted. The current guidance from SIG API Machinery is to use a separate etcd instance for your server, which increases your operational burden (installation, upgrading, data backups, disaster recovery).

Modifying the Kubernetes source code makes sense only if you’re adding a new “core” type, where core is defined as something so crucial that you could not otherwise have a functional cluster without it. The bar for getting a new core API type approved is high, so one of the other options is most likely your best bet.

Defining CRDs

Let’s start by defining a simple CRD for widgets with the following YAML:

Save the file as widgets-crd.yaml, then run kubectl create -f widgets-crd.yaml to register your CRD with the cluster. Wait a couple of seconds to give the system time to process your new CRD, and then check its status via kubectl get crd/widgets.example.heptio.com -o yaml. Your output should look similar to this:

Note: it is possible to define and register your CRD in your code, but having it in an external file is a better approach — you don’t have to give your code permission to create or delete CRDs. It’s also easier to deregister the CRD this way — run kubectl delete -f widget-crd.yaml to delete the CRD along with any instances of the custom resource.

You can now get, list, create, update, and delete widgets in the example.heptio.com API group. The fields that make up a widget are entirely up to you. For example, you could write:

There’s much more to working with CRDs, but this post gives you the big picture and shows you how the basics work. The next steps I’d recommend are learning how to codify API types using Go structs, and then using the same code generation tooling that Kubernetes uses to create deepcopy functions, clientsets, listers, and shared informers. You can learn all about these in Stefan Schimanski’s post, Kubernetes Deep Dive: Code Generation for CustomResources.