Extending Kubernetes

,

Intro

Recently, I built a Postgres controller for Kubernetes using Kubebuilder. The controller handles some basic chores for databases under controller management. This involves CRDs and implementing a controller that ensures the desired state of the databases are met. With the Postgres controller, it’s possible to automate tasks such as creating databases, managing users, etc.

While I won’t be showing any code examples, let’s talk about the concepts of extending Kubernetes.

Platform

Before talking about extending Kubernetes, it’s worth talking about platforms. In the modern software engineering landscape, a platform refers to a set of tools, technologies, and services that are designed to support your software development lifecycle. Platforms provide a way to abstract away the complexity of infrastructure and provide developers with a set of tools that make it easier to build and deploy applications; providing a golden path. This allows developers to focus on the fun business logic, rather than focusing on how their app should run.

Kubernetes is a platform, for building platforms.

CRDs

By defining custom resource definitions (CRDs), you can create an abstraction of all sorts of things that can be managed by Kubernetes. This in turn, creates new K8s APIs. This makes it easier to manage and deploy complex apps or services on the platform, especially as more and more moving parts are being mixed into modern apps.

This is what makes Kubernetes so powerful for building platforms. It gives you a foundation to build cloud-native applications that can interact with all sorts of services like S3, Elasticsearch, message queues, and more.

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: systems.mikemiller.tech/v1alpha1
kind: Database
metadata:
  name: foo-db
  namespace: infra
spec:
  replication: true
  size: 120Gi
  version: 14
  credentials:
    username: foo

This is what a CR could look like, the sky’s the limit

Controllers

So we’ve covered CRDs and APIs, but what about controllers (or operators) and reconcilers? What do they actually do?

A controller in Kubernetes is a control loop that manages the state of a particular resource. It watches for changes in the desired state of a resource and takes action to make the current state match the desired state. A reconciler is a part of the controller that compares the desired state of the resource with the actual state of the resource and takes action to reconcile the differences. The reconciler is responsible for creating, updating, or deleting the resources that the controller manages.

For example in the Postgres controller, the controller watches for changes in the desired state of a database, such as modifying the configuration of an existing database or creating a new database. The reconciler will take the necessary action to ensure that the current state of the database matches the desired state that we have defined in the CR.

With controllers we can also implement validating webhooks, that can help us ensure resources are valid before they are applied to the cluster. Similarly, we can also implement mutating webhooks with controllers, which can modify resources before they’re applied to the cluster. This can be helpful if you’re injecting sidecars, think Istio or Vault style sidecars.

Final Thoughts

I had a great time working on the Postgres controller. It gave me a chance to deep dive into the inner workings of Kubernetes. Not only did I get to have some fun writing Go, but I learned some important concepts about how Kubernetes APIs are designed and implemented. What made this project especially rewarding was how simple and elegant the solution turned out to be.

In addition to building custom controllers and CRDs, another way to extend Kubernetes is by writing Istio Wasm (WebAssembly) plugins to extend service mesh capabilities, or by writing Prometheus exporters to monitor custom metrics. I plan on writing about Istio Wasm plugins soon, stay tuned.

I can’t stress how powerful something like this is in a modern platform! Extending Kubernetes is a big deal for modern app development; it lets you build cloud-native apps that work with tons of services in a single, unified API. Get out there and try it yourself!