DataStax Developer Blog

Isolating OpsCenter Performance Data

By Mike Bulman -  June 4, 2012 | 7 Comments

Update: Since writing this post we have added first class support for storing OpsCenter data in a separate, dedicated cluster. Read more here: Storing collection data on a different cluster

One of the core features of DataStax OpsCenter is that it monitors all of your nodes and keeps track of historical metrics. We made the decision early on to leverage the DataStax Enterprise or Cassandra cluster being monitored to store all of this data. By doing this, your data is automatically replicated, and you don’t have to worry about scaling a separate storage system as your cluster grows. You can read more details about how we store this data in last week’s post.

While we’ve done a lot of work to minimize the performance and storage impacts of this data on the cluster — an average cluster will never use more than 1GB per node — we certainly understand why some users may want to isolate OpsCenter data from their production data. Whether it’s strictly for storage capacity purposes, or the need to isolate reads/writes for performance or tracking reasons, I’m going to show you how to store your OpsCenter data on dedicated nodes.

The approach I’m going to describe involves making your cluster data center aware, and utilizing replication settings in each data center to control where data is stored. For the purposes of this post I’m going to utilize the PropertyFileSnitch that is shipped with Cassandra, but the concept can be applied to any data center aware snitch.

For this example, we’ll create two data centers: Production and OpsCenter. First, use this PropertyFileSnitch reference to configure your data centers and nodes. Next, you’ll need to configure your keyspaces to write to the proper nodes. You can do this via the Cassandra CLI:

Note: This configuration change should be made before running OpsCenter for the first time to avoid losing any OpsCenter specific data that may have already been stored.

USE OpsCenter;
UPDATE KEYSPACE OpsCenter WITH placement_strategy='NetworkTopologyStrategy' and strategy_options={OpsCenter:2};

USE Keyspace1;
UPDATE KEYSPACE Keyspace1 WITH placement_strategy='NetworkTopologyStrategy' and strategy_options={Production:3};

USE MyKeyspace2;
UPDATE KEYSPACE MyKeyspace2 WITH placement_strategy='NetworkTopologyStrategy' and strategy_options={Production:2};

And you’re all set. Multiple copies of each type of data will be stored on the nodes you’ve specified for their respective data centers.

 

Tradeoffs

While this approach will accomplish the intended goal, there are some tradeoffs. Since OpsCenter data is no longer stored on the nodes with your production data, as the number of nodes in your Production data center grows, you will have to manually monitor and scale the set of OpsCenter nodes to accommodate growing data sizes.

Also, for any newly created keyspaces you will need to be sure to properly setup their replication strategy options before any data is inserted. If this isn’t done you will end up storing production data on your OpsCenter nodes.

Feedback

We’re constantly trying to learn more about use cases for isolating the data OpsCenter collects, and would love to hear about why that might be something you require, and whether this solution will or will not work for you. So please don’t hesitate to tell us more in the comments.



Comments

  1. Sankalp says:

    It is going to give you this error
    [default@OpsCenter] UPDATE KEYSPACE OpsCenter with strategy_options = [{OpsCenter:2}];
    WARNING: [{}] strategy_options syntax is deprecated, please use {}
    SimpleStrategy requires a replication_factor strategy option.

  2. Mike Bulman says:

    Thanks Sankalp. I’ve updated the post to convert the keyspaces to use NetworkTopologyStrategy at the same time.

  3. Sankalp says:

    I think it will not work by updating the keyspace and running repair.
    When you change OpsCenter keyspace from
    UPDATE KEYSPACE OpsCenter with placement_strategy = ‘org.apache.cassandra.locator.SimpleStrategy’ and strategy_options = [{replication_factor:2}];
    to
    UPDATE KEYSPACE OpsCenter WITH placement_strategy=’NetworkTopologyStrategy’ and strategy_options={OpsCenter:2};

    Now when you run the repair, how will it get the data from the old nodes? Repair task will try to get consensus of data between 2 nodes in OpcCenter datacenter.
    So I think it will not work.

  4. Sankalp says:

    What I mean to say is that once you update the replication, there are no overlapping nodes. As in all nodes are new now. So from where will it pull the data from during repair.

  5. Mike Bulman says:

    Sankalp, that is correct. I’ve updated the note. There are ways (although fairly complicated) of migrating the data, but it would be much easier to make this change when first setting up OpsCenter.

    Thanks again.

  6. Sankalp says:

    It looks like a chicken and egg problem. Till you don’t start Opscenter, it does not create the OpsCenter keyspace for you. And if you start it, you cannot move from it. Can you put the schema to create the keyspace and all CFs.

    I ran OpsCenter(just the server and not agents) and then stopped it. Then I truncated all the Cfs in OpsCenter. Now when I update the keyspace to use a new DC, it only shows me nodes of that DC.
    Then I again tried updating it to use another Dc and it then started showing me nodes of that DC.

  7. Mike Bulman says:

    Correct. This is a limitation of the describe_ring call in Cassandra. You will only see nodes in OpsCenter that have a keyspace configured to write to them. OpsCenter will combine nodes from all keyspaces, so if you create a new keyspace configured to replicate to your other nodes, you should see all nodes.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>