Unbalanced load « DataStax Support Forums

DataStax Support Forums » Apache Cassandra

Unbalanced load

(9 posts) (3 voices)

Started 3 months ago by Dimitris Gkoris
Latest reply from nickmbailey
This topic is resolved

Tags:

Dimitris Gkoris
Member

Hello,

I am running 5 Cassandra servers for about 4 months with no problems, but I just noticed the following when I run the command: nodetool -host localhost ring

Address DC Rack Status State Load Owns Token
136112946768375385385349842972707284582
XX.211.156.XXX 211 156 Up Normal 132.97 GB 20.00% 0
XX.211.166.XXX 211 166 Up Normal 64.7 GB 20.00% 34028236692093846346337460743176821145
XX.211.166.XXX 211 166 Up Normal 71.31 GB 20.00% 68056473384187692692674921486353642291
XX.211.162.XXX 211 162 Up Normal 70.96 GB 20.00% 102084710076281539039012382229530463436
XX.211.165.XXX 211 165 Up Normal 100.11 GB 20.00% 136112946768375385385349842972707284582

The way I set up the servers was according to the tutorial: http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster (for a Single Data Center)

It seems that the load is not balanced; I remember that each node had equal load in the beginning. Could you please suggest what might have happened and how to resolve this? I tried to use the nodetool -host localhost repair several times but it doesn't seem to help.

Many thanks. :)

Posted 3 months ago #
cathy@datastax.com
Member

If you are running the RandomPartitioner then you can just run repair and cleanup on all nodes. The owns % represents the amount of data owned by a node but does not account for data which may need to be cleaned up, which would be the case if you added nodes and had to move tokens. Please see: http://www.datastax.com/docs/1.0/operations/cluster_management#adding-nodes-to-a-cluster

Posted 3 months ago #
Dimitris Gkoris
Member

Hello Cathy and sorry for the late response.

I tried the cleanup already (I tried again just now) and I read the article you told me and everything is correct except the auto_bootstrap which I have it to false. Should I set it to true? I don't think I need because I never added or moved a node. My initial setup was 5 servers from the very beginning.

Many thanks for your help.

Posted 2 months ago #
Dimitris Gkoris
Member

I just restarted each of the servers and the problem is solved. Many thanks.

Posted 2 months ago #
nickmbailey
Administrator

Dimitris, glad to hear your issue is resolved.

You should also take a look at the documentation concerning token selection when you have multiple racks. It was recently updated to include extra information for that use case: http://www.datastax.com/docs/1.0/install/cluster_init#calculating-tokens

Briefly, it is best if your dc contains the same number of nodes in each rack and that you alternate racks with token selection. It looks like your ring could use some adjustment in this regard.

Posted 2 months ago #
Dimitris Gkoris
Member

Thanks a lot Nick,

However my understanding is that I am having the following 4 racks: 156, 162, 165, and 166 because of the IP adresses of my servers, right? Since I have 2 nodes with IPs: XX.211.166.XX1 and XX.211.166.XX2, I end up with 2 nodes using the same rack 166.

This tutorial recommends the following setup: rack1, rack2, rack3, rack1, rack2, rack3, and so on (having the same number of nodes in each rack).

The problem is that I am using a cloud server provider to host my servers. Since the IP address of each server is given by the cloud server provider, these IPs are being generated and allocated randomly every time I create a new server through the control panel. Therefore, I would have to create lots of cloud servers hoping that I will end up with the following IPs: XX.211.RC1.XX1, XX.211.RC2.XX2, XX.211.RC1.XX3, XX.211.RC2.XX4 and XX.211.RC1.XX5 which means that I will have to be very lucky. Also, with a 5 server setup I will never have the same number of nodes in each rack unless I use only 1 rack which is almost impossible (for the reason I described above). I read somewhere that if I want to perform quorum operations on my database, it is more efficient to have an odd number of servers (3, 5, 7 and so on). Finally, I have this service running for a few months now and it's too risky to make such a big change now.

Please let me know if I am wrong and how I could possibly have a better configuration with the current IP addresses.

Best regards,
Dimitris.

Posted 2 months ago #
nickmbailey
Administrator

Dimitris,

Are you using SimpleStrategy or NetworkTopoloyStrategy as the replication strategy for your keyspaces? If you are using SimpleStrategy then the rack of a machine doesn't matter and you can ignore the rack property of your nodes. If you are using NetworkTopologyStrategy, then the rack placement of a machine is considered and a ring like yours could cause imbalances.

Your point about the assigned ips from your cloud provider is valid. Since it is impossible to know if machines are in the same rack when in the cloud, generally you should use a snitch that doesn't determine rack info, such as PropertyFileSnitch or SimpleSnitch. That will allow your machines to all have the same rack.

Nick

Posted 2 months ago #
Dimitris Gkoris
Member

Hi Nick, yes I am using SimpleStrategy, but I am using RackInferringSnitch which is perhaps why my ring was unbalanced. However, it is ok since I restarted all of the 5 servers a few days ago.

I admit that I blindly followed your tutorial at http://www.datastax.com/docs/1.0/install/cluster_init#initializing-a-multi-node-or-multi-data-center-cluster without thinking too much about it. If the imbalances occur again then I will try to change the endpoint_snitch to SimpleSnitch and see how it works. For now, I will leave it as it is, because I am curious to see if I get the same problem with the current settings after a few months again.

Again, many thanks for your help and valuable knowledge.
Dimitris.

Posted 2 months ago #
nickmbailey
Administrator

Dimitris,

Just so you know,since you are using SimpleStrategy then the racks in your cluster don't matter. As long as you continue using SimpleStrategy you the racks won't cause any imbalance problems. If you do ever use NetworkTopologyStrategy though, you may need to change your snitch so that all nodes are in the same rack.

Posted 2 months ago #

RSS feed for this topic

Reply

You must log in to post.