DataStax Support Forums » Topic: PropertyFile vs. BriskSnitch

DataStax Support Forums » Topic: PropertyFile vs. BriskSnitch http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 09:12:34 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php joaquin on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-226 Fri, 24 Jun 2011 00:23:49 +0000 joaquin 226@http://www.datastax.com/support-forums/ Glad to hear! blueplastic on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-225 Fri, 24 Jun 2011 00:00:37 +0000 blueplastic 225@http://www.datastax.com/support-forums/ Cleaned out /var/lib/cassandra and I'm all good. Thanks! joaquin on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-224 Thu, 23 Jun 2011 21:51:24 +0000 joaquin 224@http://www.datastax.com/support-forums/ The only way to change tokens after they've been set is using a nodetool move <token> or clearing /var/lib/cassandra. The latter would be better in your case since there is no data, right? This command will clear all data, if any. It seems as though you cannot move a token without having another node staying stable, which makes sense since the datacenter would be unusable for the time being. blueplastic on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-223 Thu, 23 Jun 2011 21:41:18 +0000 blueplastic 223@http://www.datastax.com/support-forums/ Cool, got it. Also, for anyone interested in this topic of token placements across multiple data centers, I recommend checking out this discussion thread on the Apache Cassandra Mailing list: <a href="http://www.mail-archive.com/user@cassandra.apache.org/msg14223.html" rel="nofollow">http://www.mail-archive.com/user@cassandra.apache.org/msg14223.html</a> In my 3-node example above, the token placement should actually be: Node 0: 0 Node 1: 85070591730234615865843651857942052864 DC2: Node 0: 42535295865117307932921825928971026432 That's actually how I have it defined in my YAML file, but I just realized that the cluster is stuck with the old token ranges. I tried stopping Cassandra on all nodes and restarting it again, but the nodetool ring is still showing a stale configuration. So, to change existing token ranges, is it fine to just stop Cassandra across the cluster, edit the YAML file and restart Cassandra? This doesn't seem to work. So, I changed the YAML file settings back to the bad/faulty ranges and then restarted Cassandra. nodetool ring still showed the bad settings, which is what I expected. Then I tried nodetool move to change the token range on node 2 (DC1:RAC2) from 56713727820156410577229101238628035242 to 85070591730234615865843651857942052864. But I got this error: ubuntu@ip-10-68-59-87:~/briskb2/resources/cassandra$ bin/nodetool -h 10.198.126.193 move 85070591730234615865843651857942052864 Exception in thread "main" java.lang.IllegalStateException: datacenter (Brisk) has no more endpoints, (1) replicas still needed at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:124) at org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:207) at org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:234) at org.apache.cassandra.service.StorageService.getRangesForEndpoint(StorageService.java:1563) at org.apache.cassandra.service.StorageService.move(StorageService.java:1852) at org.apache.cassandra.service.StorageService.move(StorageService.java:1803) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) joaquin on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-222 Thu, 23 Jun 2011 20:56:29 +0000 joaquin 222@http://www.datastax.com/support-forums/ Each DC has it's own set of replication families. Yes, with the network topology strategy will allow you to replicate writes that way. Correct, this is where the confusion has been coming from. We'll start using the phrase "replication families." DC1 has two nodes in it's replication family. DC2 has one node. What you would want to see would be: DC1 at tokens 0 and 50, with DC2 at any token other than 0 and 50. The node on DC2 will still own 100% of the ring. For visual purposes, I would put DC2's node at token 25 or 75, but it doesn't change the fact that if you write data to DC2 it will ALWAYS fall on that node. What is wrong with your current ring is the fact that on the DC1 replication family, node1 will own 66% of all tokens in that replication family, while node2 will own 33%. This is going to cause more stress to node1. Again, this is because they are two different replication families. Again, there are some tools, like nodetool ring and OpsCenter, on it's main view, that don't yet show the full benefit of having two DCs visually. Let me know if I'm still missing clarification! :) blueplastic on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-221 Thu, 23 Jun 2011 20:36:22 +0000 blueplastic 221@http://www.datastax.com/support-forums/ Err, Joaquin, can you clarify this? "So, when you say you want the 4 Brisk nodes to have a separate ring, you can do that, but also visualize them being a separate cluster. You can then configure your client to always write to both the vanilla cluster and the tt cluster..." By separate ring, I just meant that since the 4 Brisk nodes will be in DC2, they automatically will join a different ring, right? I thought each DC had its own ring and a ring can't span across DCs. So, if the 8 vanillas and the 4 brisk/tts join two different rings, I should still be able to write to the 8 vanillas and have the Network Topology Strategy replicate the write to the 4 brisk nodes? So my client shouldn't need to write twice, once to the 8 vanillas and once to the 4 brisks. Here's some output from a little test cluster: ubuntu@ip-10-68-x-x:~/briskb2/resources/cassandra$ bin/nodetool -h localhost ring Address DC Rack Status State Load Owns Token 113427455640312821154458202477256070485 10.68.x.x DC1 RAC1 Up Normal 29.87 KB 33.33% 0 10.198.x.x DC1 RAC2 Up Normal 34.17 KB 33.33% 56713727820156410577229101238628035242 10.204.x.x DC2 RAC1 Up Normal 25.51 KB 33.33% 113427455640312821154458202477256070485 What doesn't make sense to me is that since the last node is in DC2 by itself, shouldn't it own 100% of the ring (in DC2)? And the other two nodes in DC1 should own 50% each. In the above example, 10.68.x.x and 10.204.x.x are seeds. 10.204.x.x was started with 'brisk/cassandra -t', but the other two were started as just vanilla cassandra (no -t). joaquin on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-220 Wed, 22 Jun 2011 19:16:58 +0000 joaquin 220@http://www.datastax.com/support-forums/ Hello again! Yes, that is totally possible. Your outline for it won't work as a efficiently however. Here's why: So "Brisk" as a whole is a package. It contains any number combination of vanilla Cassandra nodes and Brisk nodes, but in order to simplify things, we will call them vanilla nodes and tt (task tracker) nodes. Vanilla nodes contain just Cassandra. TT nodes contain Cassandra and also run task trackers. In order to help explain what exactly the snitch does, I'll go through and explain why we use a DCs in Brisk. So if we connect 12 nodes via SimpleSnitch, we will form one ring. This one ring will be split up into 12 spots. We will then go and change the snitch to be EC2Snitch. This will keep the same 12 nodes on the same 12 spots of the token, but since we have the EC2Snitch, it will automatically set 4 of those nodes to be on another "DC" as according to Cassandra since.. they are on another DC. With the cluster set like this, we are able to easily set different options as far as data replication/safe-keeping is concerned. For example, every time data gets written to the east coast, send one copy to the west coast. When the east coast has a power outage, we have all the data still operable on the west coast. So, with the BriskSnitch, given 12 nodes, all running vanilla nodes. All nodes will appear in the same cluster. If you were to not start all 12 nodes as vanilla nodes, and instead did 8 vanilla and 4 tt nodes, we would get 8 nodes in one "DC" and 4 in another "DC" because Cassandra sees them as such, not because they actually ARE in physical DCs. But, if they were in different DCs then the result would be exactly the same. Because we tricked Cassandra however, we are able to setup the cluster easily to always send data to the Brisk DC as well. This way both parts of the same cluster, have the same information. The clients will read and write to the vanilla nodes and all the heavy analytics will fall on the tt nodes and all the while, we don't have to make sure the tt nodes have the most up to date information, since this is built-in power available at the multiple DC level. So, when you say you want the 4 Brisk nodes to have a separate ring, you can do that, but also visualize them being a separate cluster. You can then configure your client to always write to both the vanilla cluster and the tt cluster, this way you can run analytics with real-time data. Or you can follow the above setup, to take the load off the client on replicating to two different clusters since this is already an innate function of Cassandra DataCenter setup. So as far as your concern for having a production ready HA/DR, this is exactly what you would have if 4 nodes were on a physically different datacenter. The BriskSnitch just used Cassandra's powerful innate HA/DR and watered it down for innate replication management, but it's still the same powerful tool as soon as physical datacenters are used. See <a href="http://www.datastax.com/docs/0.8/operations/datacenter" rel="nofollow">http://www.datastax.com/docs/0.8/operations/datacenter</a> for more information. Quick note on tokens: You want the tokens to be equally split on each DC since each DC will run it's own replication algorithms. If on every conflict, you offset one ring by one until you have no conflicts, then this will work 100% fine. But if you ever plan on using OpsCenter, it's just more visually pleasing to see the ring nicely split altogether. Actually, for a 2 DC ring splitting algorithm see: <a href="https://github.com/riptano/BriskClusterAMI/blob/master/tokentool.py" rel="nofollow">https://github.com/riptano/BriskClusterAMI/blob/master/tokentool.py</a> blueplastic on "PropertyFile vs. BriskSnitch" http://www.datastax.com/support-forums/topic/propertyfile-vs-brisksnitch#post-219 Wed, 22 Jun 2011 17:46:25 +0000 blueplastic 219@http://www.datastax.com/support-forums/ Is it possible to use PropertyFile Snitch with a Brisk Cluster? We want to have 8 vanilla Cassandra nodes in one Availability Zone and 4 Brisk nodes in another Availability Zone (all in same region). When doing this, we'd like to put the 8 Cassandra nodes in DC1:RAC1 and DC1:RAC2 and the 4 Brisk nodes in DC2:RAC1. Also, the 8 Cassandra nodes would join their own ring and the 4 Brisk nodes would have a separate ring. (so, I'll run the Python token generator twice, once with 8 and once with 4) Would this work fine with Brisk? I know using BriskSnitch with token placement alternates between Cassandra and Brisk nodes is the solution recommended in the DataStax documentation, but we wanted to explore a more production-ready (HA/DR) type of configuration. BriskSnitch is intended for mixed-workload Brisk clusters located in one physical data center. We want to explore putting Brisk in a separate data center.