Conventional wisdom says you can’t run a database in a container because “Containers are stateless!” or “Databases are pointless without state!” However, this is not true any longer. We’ve been missing the right technology to support database running in a container.
Then came Kubernetes 1.5, which includes the new StatefulSet API object (in older versions, StatefulSet was known as PetSet). With StatefulSets, Kubernetes makes it easier to run stateful workloads such as Couchbase Server, a NoSQL database.
Stateful Containers
Kubernetes StatefulSets gives you a set of resources to deal with stateful containers, such as: volumes, stable network ids, ordinal indexes from 0 to N, etc. Volumes are one of the key features that allow us to run stateful applications on top of Kubernetes; let’s see the two main types currently supported:
Ephemeral storages volumes
The behavior of ephemeral storages is different than what you are used to in Docker. In Kubernetes, the volume outlives any containers that run within the Pod, and the data is preserved across container restarts. But if the Pod getskilled, the volume is automatically removed.
Persistent storage volumes
In a persistent storage, as the name suggests, the data lifetime is independent of the Pod’s lifetime. So, even when the Pod dies or is moved to another node, that data will still persist until it is explicitly deleted by the user. In those kinds of volumes, the data is typically stored remotely.
We are looking forward for Kubernetes to support Local Persistent Storages as it will definitely be the best fit for running databases, but in the meantime, we use ephemeral storages by default for Couchbase Server. At this point, you might wonder why we are using ephemeral storages instead of the persistent ones. Not surprisingly, there are many reasons for that:
-
Ephemeral storages are faster and cheaper than persistent; it would require more infrastructure/networking to use persistent storages as you need to send the data back and forth
-
K8s 1.9 introduced Raw Block Support, which allows you to access physical disks in your VM instance to use it in your application
-
Maintain networked storage systems is not trivial
-
You can always try to reboot the container first instead of killing the whole Pod: Ex (kubectl exec POD_NAME -c CONTAINER_NAME reboot)
-
You can configure the database to automatically replicate your data, so even if N Pods dies, no data will be lost
-
Part of K8s job is to run Pods in different racks to avoid massive failures
However, there are a few scenarios where using Remote Persistent Storages would be worth the extra latency cost, like in massive databases for instance, when the rebalancing process takes several minutes to finish. That is why we also will add support for Remote Persistent Storages for Couchbase Server.
One of the downsides of Statefulsets is the limited management. This is why we decided to extend the Kubernetes API through the use of a Custom Resource Definition (CRD), which allows us to create a custom native resource in Kubernetes similar to a StatefulSet or a Deployment, but designed specifically for managing Couchbase instances.
Great! So, with StatefulSets/CRDs we have all the hardware operations arranged, but there is just a “small” thing missing here: what about the state of the application itself? In a database, for instance, adding a new node to the cluster is not nearly enough. You would still be required to trigger some processes, such as rebalancing to move/replicate some of the data to the newly added node to make it fully operational. That is exactly why K8s Operators joined the game.
Kubernetes Operators
Kubernetes 1.7 has added an important feature called Custom Controllers. In summary, it enables developers to extend and add new functionalities, replace existing ones (like replacing kube-proxy for instance), and of course, automate administration tasks as if they were a native Kubernetes component.
An Operator is nothing more than a set of application-specific custom controllers. So, why is it a game changer? Well, controllers have direct access to Kubernetes API, which means they can monitor the cluster, change pods/services, scale up/down, and call endpoints of the running applications, all according to custom rules written inside those controllers.
To illustrate this behavior, let’s see how Couchbase’s Operator works when a Pod gets killed:
As you can see in the figure above, the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve a desired state. This reconciliation process is all over the place in K8s. But not all actions are equal; in our example, we have two distinct categories:
-
Infrastructure – add a new node to the cluster: The operator requests via Kubernetes API to launch a new Pod running Couchbase Server.
-
Domain Specific – add node to cluster/ trigger data rebalancing: The operator knows how Couchbase works and calls the correct rest endpoint to add the new node to the cluster and trigger data rebalancing.
That is the real power of Operators: they allow you to write an application to fully manage another, and guess which kind stateful applications are the hardest to manage? You are right: Databases.
Developers have always expected databases to work out-of-the-box, when in fact, they historically are exactly the opposite. We even have a specific name for the person responsible for taking care of the database: our beloved DBAs.
Couchbase’s Operator was created as an effort change this scenario and make databases easy to manage without locking you to a specific cloud vendor. Currently, it supports automated cluster provisioning, elastic scalability, auto recovery, logging and access to the web console, but many more features are coming in the future. If you want to read more about it, please check out this article or refer to Couchbase’s official documentation here.
I also have to mention that It is the very first official operator launched for a database, although there are already some small community projects trying to build operators for MySQL, Postgres and other databases.
The Operator’s ecosystem is growing quickly; rook for example, lets you deploy something very similar to AWS S3. The Apache Kafka operator is coming soon, and with so many other initiatives out there, we expect a major boost in the number of operators in the upcoming months now that all major cloud providers supports K8s.
Finally, Kubernetes provides a cloud-agnostic application deployment and management. It is so powerful that it might lead us to treat cloud providers almost like a commodity, as you will be able to migrate freely between them.
In the future, choosing a cloud provider could be just a matter of which one offers the best performance/cost.
As Director of Product Management at Couchbase, Anil Kumar is responsible for product strategy, development, and delivery of Couchbase Data Platform offerings. Prior to joining Couchbase, Anil spent several years working at Microsoft in the Entertainment division and most recently in the Windows and Windows Live division.
For similar topics on Kubernetes, consider attending KubeCon + CloudNativeCon EU, May 2-4, 2018 in Copenhagen, Denmark.
