Adding second node removes first node from nodetool ring

(5 posts) (2 voices)

• Started 1 year ago by
• Latest reply from
• This topic is resolved

Tags:

• cassandra
• nodetool

1. 

   Anonymous
   Unregistered

   Hi,

   I have installed cassandra on freebsd hyper-v virtual machine. The first node seems to be running ok with the following settings:
   cluster_name: 'Avargy Cluster'
   initial_token: 0
   seed_provider:
   - seeds: "192.168.0.247"
   listen_address: 192.168.0.247
   rpc_address: 192.168.0.247
   (the rest is at default settings)
   so - running "nodetool -h localhost ring" returns:
   Address DC Rack Status State Load Owns Token
   192.168.0.247 datacenter1 rack1 Up Normal 22.21 KB 100.00% 0

   So, I cloned the first machine, changed ip, host-name in /etc/rc.conf and added a dns entry. The network connection works fine. I don't think that the clone is the related to the problem - just to complete the picture...
   I have disabled the firewall so it is not in the way :-)

   Now - I updated the cassandra.yaml script on the second node:
   initial_token: 85070591730234615865843651857942052864
   seed_provider:
   - seeds: "192.168.0.247"
   listen_address: 192.168.0.248
   rpc_address: 192.168.0.248
   (the rest is as node1)

   now - after starting cassandra I run "nodetool -h localhost ring" again (on node1):
   Address DC Rack Status State Load Owns Token
   192.168.0.248 datacenter1 rack1 Up Normal 31.22 KB 100.00% 0

   So - apparently the new node "takes over"... and gets a token "0" - strange... :-(
   I expected (hoped) to see both nodes listed with each handling 50%...

   what do I do wrong?

   Thanks in advance!
   Thomas Augustinus

   Posted 1 year ago #
2. 

   nickmbailey
   Key Master

   When you say 'cloned the first machine', do you mean completely cloned? When you start the first node it will save configuration files to the cassandra system tables. If those files exist when the second machine starts up, it could cause a problem like this. Make sure all data from your cassandra data directories is deleted when the second machine is started.

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   That is very likely - I did not think of that!
   could you please guide me to the files / folders I need to delete?
   - or should I use a nodetool decomission?
   Thanks
   Thomas Augutinus

   Posted 1 year ago #
4. 

   nickmbailey
   Key Master

   The cassandra data directories are configured in your cassandra.yaml. By default they are set to '/var/lib/cassandra'. I would reccommend deleting that directory on both nodes and starting with a completely clean install.

   Posted 1 year ago #
5. 

   Anonymous
   Unregistered

   Great - it worked!
   Stopped cassandra on both nodes, deleted the data, commitlog and cache folders:
   rm -R /var/lib/cassandra/data/*
   rm -R /var/lib/cassandra/commitlog/*
   rm -R /var/lib/cassandra/saved_caches/*
   then restarted cassandra on node1, wait 30 sec, then start cassandra on node 2, wait 1 min then run nodetool -h localhost ring:
   Address DC Rack Status State Load Owns Token
   85070591730234615865843651857942052864
   192.168.0.247 datacenter1 rack1 Up Normal 9.06 KB 50.00% 0
   192.168.0.248 datacenter1 rack1 Up Normal 6.66 KB 50.00% 85070591730234615865843651857942052864

   Thank you!
   Best regards
   Thomas Augustinus

   Posted 1 year ago #

RSS feed for this topic