Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

Schemas not in agreement - Brisk mixed workload cluster.

(11 posts) (3 voices)

Started 1 year ago by agriffith
Latest reply from joaquin
This topic is not resolved

Tags:

No tags yet.

agriffith
Member

I am getting strange behavior when starting up a mixed workload cluster.

6 nodes running vanilla apache cassandra v 0.8.1
3 nodes running brisk cassandra with hadoop trackers turned on.

3 of the vanilla cassandra nodes are in the seed list, 1 of the brisk nodes is in the seed list.

The cassandra.yaml config file has been updated to use the brisksimplesnitch on all nodes.

Starting all 6 vanilla cassandra nodes appears to work correctly. However, when the first brisk node is brought online, strange things start to happen.

nodetool ring shows all nodes in the cluster running.
- if you log into the brisk node with the cassandra-cli 3 of the vanilla cassandra nodes are unreachable.
- if you log into a reachable cassandra node with the cassandra-cli all 7 nodes show up (6 vanilla and 1 brisk)
- if you log into an unreachable cassandra node with the cassandra-cli the brisk node is unreachable

2 of the unreachable vanilla cassandra nodes are seeds, and 1 is a non seed.

There is an info message in the cassandra log on these nodes:
INFO [Thread-18] 2011-07-06 22:37:56,958 IncomingTcpConnection.java (line 110) Received connection from newer protocol version. Ignorning message.

This is not showing up on the reachable nodes.

I have tried bringing up the nodes in different order. When bring up 1 vanilla and 1 brisk I get a different problem. When the brisk node starts it chooses one of the down vanilla cassandra seed nodes as the job tracker. When all vanilla cassandra nodes are up, and the brisk node is started, it chooses the brisk node as the job tracker.

Any ideas on what is causing this behavior?

Posted 1 year ago #
joaquin
Administrator

What version of Brisk are you running?

It seems as though there may be a version mismatch. Is Brisk Beta2 on the Brisk nodes?

Have the nodes previously been on any other version than 0.8.1 and Brisk Beta2?

Thanks,
Joaquin

Posted 1 year ago #
joaquin
Administrator

What version of Brisk are you running?

It seems as though there may be a version mismatch. Is Brisk Beta2 on the Brisk nodes?

Have the nodes previously been on any other version than 0.8.1 and Brisk Beta2?

Thanks,
Joaquin

Posted 1 year ago #
agriffith
Member

Brisk beta 2 for the brisk nodes. the cassandra nodes were 0.8.0 and 0.7.6 at one point but everything was removed and all data wiped, then a clean install of 0.8.1 was put on.

Posted 1 year ago #
nickmbailey
Key Master

I'm not sure that brisk beta2 is compatible with 0.8.1. The version of cassandra in brisk is pre-0.8.1. Any reason you aren't using brisk on all the nodes and just turning off hadoop on some of them?

Posted 1 year ago #
agriffith
Member

It seems that if that was the case they would all be incompatible. Right now half of them work and half don't.

Posted 1 year ago #
joaquin
Administrator

If you could try running them all with Brisk Beta2 that would be ideal. Cassandra is never supposed to be run in a mixed version state for longer than it takes to do a rolling upgrade.

With 0.8.1 and 0.8.0+ running in the same cluster, you wouldn't get a non-working cluster altogether, but one that may develop problems over time. Since we have not fully tested this Cassandra and Brisk hybrid cluster, if you could move them all to be Brisk Beta2 nodes, that would be best. In future releases we hope to include stable Cassandra versions to allow for this type of hybrid clusters.

Posted 1 year ago #
agriffith
Member

Here is problem:

We have existing cassandra system running now 0.8.1. This is a production system and is not going to change to a beta unknown cassandra version.

We want to run hive/pig analytics against data so Brisk is the choice for that. We can't get brisk and existing cluster to work properly.

Are there other options for our situation besides running a mixed-workload cluster?

Can we create new brisk cluster and replicate data from existing vanilla cluster?

Thanks for any help.

Posted 1 year ago #
joaquin
Administrator

This looks like it may be this:

https://issues.apache.org/jira/browse/CASSANDRA-2860.

If possible could you perform a full cluster restart? If not, a rolling restart may fix the issue.

Thanks,
Joaquin

Posted 1 year ago #
agriffith
Member

I have tried shutting down the nodes and bringing them back up. We have a test environment where we started from scratch and still got the error.

Posted 1 year ago #
joaquin
Administrator

Hello,

This issue may also be related: https://issues.apache.org/jira/browse/CASSANDRA-2818.

Since we recently upgraded the versioning architecture we haven't gotten all the kinks out of a hybrid cluster yet. If you could hold off until 0.8.2 releases and Brisk beta3, that would be great.

In the meantime however, you can definitely rsync the data to a new cluster and run testing on that cluster if you wish.

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.