John Sanda

<p>DataStax Kubernetes Operator for Apache Cassandra™ manages Cassandra clusters in Kubernetes. It provides configuration management, scaling Cassandra (adding and removing nodes), and error handling, and as part of <a href="https://k8ssandra.io/">K8ssandra</a> it makes the necessary configuration changes required for common operations like repair and backup and restore. This post focuses on some of the configuration management capabilities of Cass Operator through a series of examples. For some general background on Cass Operator, check out the DataStax <a href="https://docs.datastax.com/en/cass-operator/doc/cass-operator/cassOperatorTOC.html">documentation</a>.&nbsp;</p>

<p>The first set of examples do not require any advanced understanding of Cass Operator or Kubernetes in general. The advanced examples assume a deeper understanding of Kubernetes as they cover topics like init containers and StatefulSets.</p>

<p>The examples have been tested against Cass Operator 1.6.0, the latest version as of this writing. The full source of the examples can be found in the <a href="https://github.com/jsanda/cassandradatacenter-examples">cassandradatacenter-examples</a> repository on GitHub.</p>

<h2>Basic examples</h2>

<p>These examples demonstrate how to do basic configuration of Cassandra and of Kubernetes resources.</p>

<h2>Cassandra and JVM</h2>

<p>The snippet of the CassandraDatacenter manifest demonstrates how to configure <code class="language-ini">cassandra.yaml</code> and <code class="language-ini">jvm-options</code>&nbsp;for a Cassandra 3.11.10 deployment.</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  config:
    cassandra-yaml:
      authenticator: PasswordAuthenticator
      authorizer: CassandraAuthorizer
      compaction_throughput_mb_per_sec: 128
      tombstone_warn_threshold: 5000
      tombstone_failure_threshold: 200000
      unlogged_batch_across_partitions_warn_threshold: 25   
    jvm-options:
      initial_heap_size: "512m"
      max_heap_size: "512m"
      heap_size_young_generation: "256m"
      garbage_collector: CMS
      max_tenuring_threshold: 5   </code>
</pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/cassandra-jvm/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>The properties under <code class="language-ini">cassandra-yaml</code> map directly to properties in <code class="language-ini">cassandra-yaml</code>. The properties under <code class="language-ini">jvm-properties</code>&nbsp;configure heap and garbage collector settings in the <code class="language-ini">jvm.options</code>&nbsp;configuration file.</p>

<p><strong>Note: </strong>Most but not all properties in <code class="language-ini">cassandra-yaml</code> and in <code class="language-ini">jvm.options</code> are supported at this time by Cass Operator.</p>

<h2>Cluster topology</h2>

<p>Cass Operator configures cluster to use racks and <code class="language-ini">GossipingPropertyFileSnitch</code>. This example configures a multi-rack cluster spread across three availability zones:</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  size: 3
  racks:
  - name: rack1
    nodeAffinityLabels:
      topology.kubernetes.io/zone: us-east1-a
  - name: rack2
    nodeAffinityLabels:
      topology.kubernetes.io/zone: us-east1-b
  - name: rack3
    nodeAffinityLabels:
      topology.kubernetes.io/zone: us-east1-c     </code>    </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/cluster-topology/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>The CassandraDatacenter declares a three node cluster with three racks. Cass Operator makes a best effort to evenly distribute nodes across racks. In this example the racks will be balanced having one node each. Cass Operator uses <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity" target="_blank">node affinity</a> to pin each rack to a different availability zone.</p>

<p><strong>Note: </strong><code class="language-ini">topology.kubernetes.io/zone</code> is a common label that is applied to all worker nodes.The CassandraDatacenter declares a three node cluster with three racks. Cass Operator makes a best effort to evenly distribute nodes across racks. In this example the racks will be balanced having one node each. Cass Operator uses <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity">node affinity</a> to pin each rack to a different availability zone.</p>

<p><strong>Note: </strong>Your Kubernetes cluster must have at least three worker nodes with the appropriate labels for this example to work.</p>

<p>Cass Operator uses <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity">pod anti-affinity</a> by default to ensure that Cassandra pods are isolated from one another. Kubernetes will not schedule multiple Cassandra pods on the same worker node.</p>

<h2>Cassandra pod resources</h2>

<p>In general it is a good practice to specify <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/">resource requirements</a> for Kubernetes applications and services. Cassandra is no exception. A pod can have multiple containers. The Cassandra pod includes one init container and two main containers.&nbsp;</p>

<p>The <code class="language-ini">server-config-init container</code>&nbsp;generates all of the configuration files in <code class="language-ini">/etc/cassandra</code>.</p>

<p>The <code class="language-ini">cassandra&nbsp;</code>container runs Cassandra. The container runs the <a href="https://github.com/k8ssandra/management-api-for-apache-cassandra">management-api</a> as PID 1. It manages the lifecycle of Cassandra. Because it is the primary process in the container, the management-api's logs go to stdout. This means that if you execute kubectl logs &lt;cassandra-pod&gt; -c cassandra you will get back the management-api's logs and not Cassandra's.</p>

<p>&nbsp;The <code class="language-ini">server-system-logger</code>&nbsp;container is a lightweight busybox container that tails <code class="language-ini">/var/log/cassandra/system.log</code>. You can view Cassandra's logs with <code class="language-ini">/kubectl logs &lt;cassandra-pod&gt; -c server-system-logger</code>.&nbsp;</p>

<p>This example illustrates how to specify CPU and memory requirements for each of these containers.</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  resources:
    requests:
      cpu: 2
      memory: 2Gi
  configBuilderResources:
    requests:
      cpu: 1
      memory: 125Mi
    limits:
      cpu: 1
      memory: 250Mi
  systemLoggerResources:
    requests:
      cpu: 100m
      memory: 20Mi
    limits:
      cpu: 100m
      memory: 30Mi      </code>    </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/cassandra-pod-resources/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>Kubernetes will only schedule the Cassandra pods on worker nodes with enough resources to satisfy the requests.</p>

<h2>Advanced Examples</h2>

<p>The following examples are advanced for a couple reasons. First, they require more understanding of Kubernetes types and concepts like <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSets</a>, <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">init containers</a>, and <a href="https://kubernetes.io/docs/concepts/storage/volumes/">volumes</a>. Secondly, the examples touch on implementation details of Cass Operator.</p>

<p>Cass Operator creates a StatefulSet for each rack. The template property of a StatefulSet fully describes the pods that will be created.</p>

<h3>Custom pod labels</h3>

<p>Suppose you want to add custom labels to the Cassandra pods. There is no specific property like&nbsp;<code class="language-ini">podLabels</code>, in the <code class="language-ini">CassandraDatacenter</code>&nbsp;object to do this. The <code class="language-ini">podTemplateSpec</code>&nbsp;property however, makes it possible.</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:

  podTemplateSpec:
    metadata:
      labels:
        env: dev
        app.kubernetes.io/part-of: examples
        app.kubernetes.io/version: "0.47.3"
    spec:
      containers: []      </code>    </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/pod-labels/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>Cass Operator will add each of these labels to the template of the StatefulSet. This in turn means that they will be added to each of the pods.</p>

<p><strong>Note: </strong>If the containers property is omitted, then we get a validation error about a null value; thus, we set it to an empty array.Cass Operator will add each of these labels to the template of the StatefulSet. This in turn means that they will be added to each of the pods.</p>

<h2>Environment Variable</h2>

<p>This next example demonstrates how to add an environment variable to the cassandra container.&nbsp;</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:

  podTemplateSpec:
    spec:
      containers:
      - name: cassandra
        env:
        - name: HELLO
          value: WORLD</code>   </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/tree/main/env-var"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>In and of itself this may not seem particularly useful, but it actually highlights something very interesting: how Cass Operator applies merge semantics with&nbsp;<code class="language-ini">podTemplateSpec</code>. As previously mentioned, we know that Cass Operator already defines the <code class="language-ini">cassandra</code>&nbsp;container. It also defines several default environment variables for the container. The declaration in this&nbsp;<code class="language-ini">podTemplateSpec</code>&nbsp;does not replace the default&nbsp;<code class="language-ini">cassandra</code>&nbsp;container nor does it replace the default environment variables. The environment variable will be added to the list of environment variables along with the default ones.</p>

<h3>Init container</h3>

<p>Now let's take a look at how to add an init container to the Cassandra pod.</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  podTemplateSpec:
    containers: []
    initContainers:
    - name: hello
      image: busybox
      args:
      - /bin/sh
      - -c
      - echo Hello World      </code>    </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/init-container/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>Init containers run in the order declared. The hello container will run before the server-config-init container. If we want the server-config-init container to run first, we can make the following change:</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  podTemplateSpec:
    containers: []
    initContainers:
    - name: server-config-init
    - name: hello
      image: busybox
      args:
      - /bin/sh
      - -c
      - echo Hello World
      </code>    </pre>
</div>
</div>

<p>Cass Operator will run server-config-init first using its default configuration.</p>

<h2>Remote JMX</h2>

<p>This final example builds off of the previous ones and is borrowed from the <a href="https://k8ssandra.io">K8ssandra</a> project. K8ssandra deploys Cass Operator along with additional components, one of which is <a href="http://cassandra-reaper.io/">Reaper</a>. Reaper manages repair operations for Cassandra. Reaper relies on JMX to perform repair operations. Cassandra has remote JMX disabled by default; so, it has to be enabled in order for Reaper to function properly. JMX authentication has to be configured as well because Cassandra enables it when remote JMX is enabled.</p>

<p>JMX access is configured in the <code class="language-ini">cassandra/etc/cassandra/cassandra-env.sh</code>&nbsp;script. It checks to see if the <code class="language-ini">LOCAL_JMX&nbsp;</code>environment variable is set. This happens in the <code class="language-ini">cassandra&nbsp;</code>container.</p>

<p>JMX credentials are stored in <code class="language-ini">/etc/cassandra/jmxremote.password</code>. We need to add a set of credentials to that file. We can use an init container for this. There is another detail to be worked out. Cass Operator does not create a volume mount for <code class="language-ini">/etc/cassandra</code>&nbsp;by default.&nbsp;</p>

<p><br />
Cass Operator creates an <code class="language-ini">emptyDir</code>&nbsp;volume named&nbsp;<code class="language-ini">server-config</code>. The <code class="language-ini">server-config-init&nbsp;</code>init container and the cassandra container both mount the volume at <code class="language-ini">/config</code>. <code class="language-ini">server-config-init&nbsp;</code>generates configuration files and writes them into this directory. When the <code class="language-ini">cassandra</code>&nbsp;container starts it copies everything from <code class="language-ini">/config</code>&nbsp;to&nbsp;<code class="language-ini">/etc/cassandra</code>. This provides us with the necessary information needed to build the init container.</p>

<div class="code">
<div class="code__heading"><span>YAML</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
      <code class="language-ini">
spec:
  podTemplateSpec:
    containers:
    - name: cassandra
      env:
      - name: LOCAL_JMX
        value: no
    initContainers:
    - name: server-config-init
    - name: jmx-credentials
      image: busybox
      env:
      - name: JMX_USERNAME
        valueFrom:
          secretKeyRef:
            name: jmx-credentials
            key: username
      - name: JMX_PASSWORD
        valueFrom:
          secretKeyRef:
            name: jmx-credentials
            key: password
      args:
      - /bin/sh
      - -c
      - echo "$JMX_USERNAME $JMX_PASSWORD" &gt; /config/jmxremote.password
      volumeMounts:
      - name: server-config
      - mountPath: /config      </code>    </pre>
</div>
</div>

<p><a href="https://github.com/jsanda/cassandradatacenter-examples/blob/main/remote-jmx/cassdc.yaml"><span><span><span><span><span><span><span><span>view source</span></span></span></span></span></span></span></span></a></p>

<p>Try to run <code class="language-ini">nodetool</code>&nbsp;without credentials using <code class="language-ini">kubectl exec -it &lt;cassandra-pod&gt; -c cassandra -- nodetool status</code>. It should fail with a&nbsp;<code class="language-ini">SecurityException</code>&nbsp;that says credentials are required. It should succeed if you run <code class="language-ini">kubectl exec -it &lt;cassandra-pod&gt; -c cassandra -- nodetool -u cassandra -pw cassandra&nbsp;status</code>.</p>

<h2>Summing up</h2>

<p>Cass Operator provides a variety of options to configure Cassandra, the JVM, and the StatefulSets that it generates. When you need to configure something that Cass Operator does not expose or when you need something more advanced like an init container with additional volumes, podTemplateSpec offers tremendous flexibility. That flexibility comes with risks, though. The remote JMX example is entirely dependent on implementation details of Cass Operator. It would be a nice enhancement for Cass Operator to enable you to configure things like init containers and sidecar containers without being tightly coupled to implementation details.</p>


DataStax Kubernetes Operator for Apache Cassandra™ manages Cassandra clusters in Kubernetes. It provides configuration management, scaling Cassandra (adding and removing nodes), and error handling, and as part of 

 it makes the necessary configuration changes required for common operations like repair and backup and restore. This post focuses on some of the configuration management capabilities of Cass Operator through a series of examples. For some general background on Cass Operator, check out the DataStax 

The first set of examples do not require any advanced understanding of Cass Operator or Kubernetes in general. The advanced examples assume a deeper understanding of Kubernetes as they cover topics like init containers and StatefulSets.

The examples have been tested against Cass Operator 1.6.0, the latest version as of this writing. The full source of the examples can be found in the 

These examples demonstrate how to do basic configuration of Cassandra and of Kubernetes resources.

The snippet of the CassandraDatacenter manifest demonstrates how to configure 

 spec: config: cassandra-yaml: authenticator: PasswordAuthenticator authorizer: CassandraAuthorizer compaction_throughput_mb_per_sec: 128 tombstone_warn_threshold: 5000 tombstone_failure_threshold: 200000 unlogged_batch_across_partitions_warn_threshold: 25 jvm-options: initial_heap_size: "512m" max_heap_size: "512m" heap_size_young_generation: "256m" garbage_collector: CMS max_tenuring_threshold: 5 

 configure heap and garbage collector settings in the 

 are supported at this time by Cass Operator.

Cass Operator configures cluster to use racks and 

. This example configures a multi-rack cluster spread across three availability zones:

 spec: size: 3 racks: - name: rack1 nodeAffinityLabels: topology.kubernetes.io/zone: us-east1-a - name: rack2 nodeAffinityLabels: topology.kubernetes.io/zone: us-east1-b - name: rack3 nodeAffinityLabels: topology.kubernetes.io/zone: us-east1-c 

The CassandraDatacenter declares a three node cluster with three racks. Cass Operator makes a best effort to evenly distribute nodes across racks. In this example the racks will be balanced having one node each. Cass Operator uses 

 to pin each rack to a different availability zone.

 is a common label that is applied to all worker nodes.The CassandraDatacenter declares a three node cluster with three racks. Cass Operator makes a best effort to evenly distribute nodes across racks. In this example the racks will be balanced having one node each. Cass Operator uses 

Your Kubernetes cluster must have at least three worker nodes with the appropriate labels for this example to work.

 by default to ensure that Cassandra pods are isolated from one another. Kubernetes will not schedule multiple Cassandra pods on the same worker node.

In general it is a good practice to specify 

 for Kubernetes applications and services. Cassandra is no exception. A pod can have multiple containers. The Cassandra pod includes one init container and two main containers.

 generates all of the configuration files in 

container runs Cassandra. The container runs the 

 as PID 1. It manages the lifecycle of Cassandra. Because it is the primary process in the container, the management-api's logs go to stdout. This means that if you execute kubectl logs <cassandra-pod> -c cassandra you will get back the management-api's logs and not Cassandra's.

 container is a lightweight busybox container that tails 

/kubectl logs <cassandra-pod> -c server-system-logger

This example illustrates how to specify CPU and memory requirements for each of these containers.

 spec: resources: requests: cpu: 2 memory: 2Gi configBuilderResources: requests: cpu: 1 memory: 125Mi limits: cpu: 1 memory: 250Mi systemLoggerResources: requests: cpu: 100m memory: 20Mi limits: cpu: 100m memory: 30Mi 

Kubernetes will only schedule the Cassandra pods on worker nodes with enough resources to satisfy the requests.

The following examples are advanced for a couple reasons. First, they require more understanding of Kubernetes types and concepts like 

. Secondly, the examples touch on implementation details of Cass Operator.

Cass Operator creates a StatefulSet for each rack. The template property of a StatefulSet fully describes the pods that will be created.

Suppose you want to add custom labels to the Cassandra pods. There is no specific property like 

 spec: podTemplateSpec: metadata: labels: env: dev app.kubernetes.io/part-of: examples app.kubernetes.io/version: "0.47.3" spec: containers: [] 

Cass Operator will add each of these labels to the template of the StatefulSet. This in turn means that they will be added to each of the pods.

If the containers property is omitted, then we get a validation error about a null value; thus, we set it to an empty array.Cass Operator will add each of these labels to the template of the StatefulSet. This in turn means that they will be added to each of the pods.

This next example demonstrates how to add an environment variable to the cassandra container.

 spec: podTemplateSpec: spec: containers: - name: cassandra env: - name: HELLO value: WORLD

In and of itself this may not seem particularly useful, but it actually highlights something very interesting: how Cass Operator applies merge semantics with 

. As previously mentioned, we know that Cass Operator already defines the 

 container. It also defines several default environment variables for the container. The declaration in this 

 container nor does it replace the default environment variables. The environment variable will be added to the list of environment variables along with the default ones.

Now let's take a look at how to add an init container to the Cassandra pod.

 spec: podTemplateSpec: containers: [] initContainers: - name: hello image: busybox args: - /bin/sh - -c - echo Hello World 

Init containers run in the order declared. The hello container will run before the server-config-init container. If we want the server-config-init container to run first, we can make the following change:

 spec: podTemplateSpec: containers: [] initContainers: - name: server-config-init - name: hello image: busybox args: - /bin/sh - -c - echo Hello World 

Cass Operator will run server-config-init first using its default configuration.

This final example builds off of the previous ones and is borrowed from the 

 project. K8ssandra deploys Cass Operator along with additional components, one of which is 

. Reaper manages repair operations for Cassandra. Reaper relies on JMX to perform repair operations. Cassandra has remote JMX disabled by default; so, it has to be enabled in order for Reaper to function properly. JMX authentication has to be configured as well because Cassandra enables it when remote JMX is enabled.

environment variable is set. This happens in the 

. We need to add a set of credentials to that file. We can use an init container for this. There is another detail to be worked out. Cass Operator does not create a volume mount for 

init container and the cassandra container both mount the volume at 

generates configuration files and writes them into this directory. When the 

 container starts it copies everything from 

. This provides us with the necessary information needed to build the init container.

 spec: podTemplateSpec: containers: - name: cassandra env: - name: LOCAL_JMX value: no initContainers: - name: server-config-init - name: jmx-credentials image: busybox env: - name: JMX_USERNAME valueFrom: secretKeyRef: name: jmx-credentials key: username - name: JMX_PASSWORD valueFrom: secretKeyRef: name: jmx-credentials key: password args: - /bin/sh - -c - echo "$JMX_USERNAME $JMX_PASSWORD" > /config/jmxremote.password volumeMounts: - name: server-config - mountPath: /config 

kubectl exec -it <cassandra-pod> -c cassandra -- nodetool status

 that says credentials are required. It should succeed if you run 

kubectl exec -it <cassandra-pod> -c cassandra -- nodetool -u cassandra -pw cassandra status

Cass Operator provides a variety of options to configure Cassandra, the JVM, and the StatefulSets that it generates. When you need to configure something that Cass Operator does not expose or when you need something more advanced like an init container with additional volumes, podTemplateSpec offers tremendous flexibility. That flexibility comes with risks, though. The remote JMX example is entirely dependent on implementation details of Cass Operator. It would be a nice enhancement for Cass Operator to enable you to configure things like init containers and sidecar containers without being tightly coupled to implementation details.

Dial C* for Operator: Unlocking Advanced Cassandra Configurations

John SandaConsultant, The Last Pickle

Discover more

Share

More Technology

Open-Source,
Scale-Out, Cloud-Native
NoSQL Database

John SandaConsultant, The Last Pickle

Discover more

Share

More Technology

Open-Source,Scale-Out, Cloud-NativeNoSQL Database

Open-Source,
Scale-Out, Cloud-Native
NoSQL Database