http://www.datastax.com/support-forums/topic/unbalanced-load Software, Support, and Training for Apache Cassandra en-US Thu, 23 May 2013 03:07:03 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/unbalanced-load#post-1235 Fri, 24 Feb 2012 20:06:04 +0000 nickmbailey 1235@http://www.datastax.com/support-forums/ <p>Dimitris,</p> <p>Just so you know,since you are using SimpleStrategy then the racks in your cluster don't matter. As long as you continue using SimpleStrategy you the racks won't cause any imbalance problems. If you do ever use NetworkTopologyStrategy though, you may need to change your snitch so that all nodes are in the same rack. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1225 Fri, 24 Feb 2012 10:52:45 +0000 Anonymous 1225@http://www.datastax.com/support-forums/ <p>Hi Nick, yes I am using SimpleStrategy, but I am using RackInferringSnitch which is perhaps why my ring was unbalanced. However, it is ok since I restarted all of the 5 servers a few days ago.</p> <p>I admit that I blindly followed your tutorial at <a href="http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster" rel="nofollow">http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster</a> without thinking too much about it. If the imbalances occur again then I will try to change the endpoint_snitch to SimpleSnitch and see how it works. For now, I will leave it as it is, because I am curious to see if I get the same problem with the current settings after a few months again.</p> <p>Again, many thanks for your help and valuable knowledge.<br /> Dimitris. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1204 Wed, 22 Feb 2012 15:57:54 +0000 nickmbailey 1204@http://www.datastax.com/support-forums/ <p>Dimitris,</p> <p>Are you using SimpleStrategy or NetworkTopoloyStrategy as the replication strategy for your keyspaces? If you are using SimpleStrategy then the rack of a machine doesn't matter and you can ignore the rack property of your nodes. If you are using NetworkTopologyStrategy, then the rack placement of a machine is considered and a ring like yours could cause imbalances. </p> <p>Your point about the assigned ips from your cloud provider is valid. Since it is impossible to know if machines are in the same rack when in the cloud, generally you should use a snitch that doesn't determine rack info, such as PropertyFileSnitch or SimpleSnitch. That will allow your machines to all have the same rack.</p> <p>Nick </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1199 Wed, 22 Feb 2012 13:09:37 +0000 Anonymous 1199@http://www.datastax.com/support-forums/ <p>Thanks a lot Nick, </p> <p>However my understanding is that I am having the following 4 racks: 156, 162, 165, and 166 because of the IP adresses of my servers, right? Since I have 2 nodes with IPs: XX.211.166.XX1 and XX.211.166.XX2, I end up with 2 nodes using the same rack 166.</p> <p>This tutorial recommends the following setup: rack1, rack2, rack3, rack1, rack2, rack3, and so on (having the same number of nodes in each rack). </p> <p>The problem is that I am using a cloud server provider to host my servers. Since the IP address of each server is given by the cloud server provider, these IPs are being generated and allocated randomly every time I create a new server through the control panel. Therefore, I would have to create lots of cloud servers hoping that I will end up with the following IPs: XX.211.RC1.XX1, XX.211.RC2.XX2, XX.211.RC1.XX3, XX.211.RC2.XX4 and XX.211.RC1.XX5 which means that I will have to be very lucky. Also, with a 5 server setup I will never have the same number of nodes in each rack unless I use only 1 rack which is almost impossible (for the reason I described above). I read somewhere that if I want to perform quorum operations on my database, it is more efficient to have an odd number of servers (3, 5, 7 and so on). Finally, I have this service running for a few months now and it's too risky to make such a big change now.</p> <p>Please let me know if I am wrong and how I could possibly have a better configuration with the current IP addresses.</p> <p>Best regards,<br /> Dimitris. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1192 Tue, 21 Feb 2012 19:05:12 +0000 nickmbailey 1192@http://www.datastax.com/support-forums/ <p>Dimitris, glad to hear your issue is resolved.</p> <p>You should also take a look at the documentation concerning token selection when you have multiple racks. It was recently updated to include extra information for that use case: <a href="http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens" rel="nofollow">http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens</a></p> <p>Briefly, it is best if your dc contains the same number of nodes in each rack and that you alternate racks with token selection. It looks like your ring could use some adjustment in this regard. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1191 Tue, 21 Feb 2012 17:59:12 +0000 Anonymous 1191@http://www.datastax.com/support-forums/ <p>I just restarted each of the servers and the problem is solved. Many thanks. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1189 Tue, 21 Feb 2012 12:43:19 +0000 Anonymous 1189@http://www.datastax.com/support-forums/ <p>Hello Cathy and sorry for the late response.</p> <p>I tried the cleanup already (I tried again just now) and I read the article you told me and everything is correct except the auto_bootstrap which I have it to false. Should I set it to true? I don't think I need because I never added or moved a node. My initial setup was 5 servers from the very beginning.</p> <p>Many thanks for your help. </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1110 Thu, 09 Feb 2012 03:04:38 +0000 cathy@datastax.com 1110@http://www.datastax.com/support-forums/ <p>If you are running the RandomPartitioner then you can just run repair and cleanup on all nodes. The owns % represents the amount of data owned by a node but does not account for data which may need to be cleaned up, which would be the case if you added nodes and had to move tokens. Please see: <a href="http://www.datastax.com/docs/1.0/operations/cluster_management#adding-nodes-to-a-cluster" rel="nofollow">http://www.datastax.com/docs/1.0/operations/cluster_management#adding-nodes-to-a-cluster</a> </p> http://www.datastax.com/support-forums/topic/unbalanced-load#post-1096 Wed, 08 Feb 2012 10:36:21 +0000 Anonymous 1096@http://www.datastax.com/support-forums/ <p>Hello,</p> <p>I am running 5 Cassandra servers for about 4 months with no problems, but I just noticed the following when I run the command: nodetool -host localhost ring</p> <p>Address DC Rack Status State Load Owns Token<br /> 136112946768375385385349842972707284582<br /> XX.211.156.XXX 211 156 Up Normal 132.97 GB 20.00% 0<br /> XX.211.166.XXX 211 166 Up Normal 64.7 GB 20.00% 34028236692093846346337460743176821145<br /> XX.211.166.XXX 211 166 Up Normal 71.31 GB 20.00% 68056473384187692692674921486353642291<br /> XX.211.162.XXX 211 162 Up Normal 70.96 GB 20.00% 102084710076281539039012382229530463436<br /> XX.211.165.XXX 211 165 Up Normal 100.11 GB 20.00% 136112946768375385385349842972707284582</p> <p>The way I set up the servers was according to the tutorial: <a href="http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster" rel="nofollow">http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster</a> (for a Single Data Center)</p> <p>It seems that the load is not balanced; I remember that each node had equal load in the beginning. Could you please suggest what might have happened and how to resolve this? I tried to use the nodetool -host localhost repair several times but it doesn't seem to help.</p> <p>Many thanks. :) </p>