DataStax Support Forums » Topic: OpsCenter Rebalance cluster

DataStax Support Forums » Topic: OpsCenter Rebalance cluster http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster Software, Support, and Training for Apache Cassandra en-US Tue, 21 May 2013 14:11:13 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php nickmbailey on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1676 Wed, 18 Apr 2012 15:36:28 +0000 nickmbailey 1676@http://www.datastax.com/support-forums/ Glad to hear that the issue is resolved! Let us know if you have any other issues or feedback. Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1658 Tue, 17 Apr 2012 19:13:57 +0000 Anonymous 1658@http://www.datastax.com/support-forums/ Hello Thobbs, Just to configm you the patched version released today work fine. Our rebalance cluster problem is fixed. Thank for this good work. The case can be closed. Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1595 Wed, 11 Apr 2012 17:45:41 +0000 Anonymous 1595@http://www.datastax.com/support-forums/ Ok, we wait for this and I keep you informed. thobbs on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1589 Wed, 11 Apr 2012 14:59:08 +0000 thobbs 1589@http://www.datastax.com/support-forums/ Ah, sorry about that. It looks like we picked the 17th as the date for the OpsCenter release that will include the fix. Try out the new version when it's released and let us know if everything is resolved. Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1588 Wed, 11 Apr 2012 14:41:58 +0000 Anonymous 1588@http://www.datastax.com/support-forums/ Unfortunately, the device name /dev/mapper/cnode03-root seems to be managed directly manager by LVM 2 (this is the default configuration for Ubuntu server 11.10) the mapping does not appear to be changed. Unless you have a procedure to do this, we will have to wait for the OpsCenter patch (or include it on the current version). thobbs on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1585 Wed, 11 Apr 2012 03:42:19 +0000 thobbs 1585@http://www.datastax.com/support-forums/ It's hard to give exact dates, but hopefully we will have a release out that includes the fix within 2 to 3 weeks. Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1564 Tue, 10 Apr 2012 17:12:14 +0000 Anonymous 1564@http://www.datastax.com/support-forums/ Many thank's Thobbs, I try the workaround and keep you informed. Do you know when the next release (with fix) is planned ? thobbs on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1554 Mon, 09 Apr 2012 17:45:31 +0000 thobbs 1554@http://www.datastax.com/support-forums/ Thanks for the output, I can definitely see what's broken here: there's a minor mismatch in how device names are escaped. I've already fixed this for the next release of OpsCenter. Unfortunately, there isn't really a good workaround for this in 1.4.1. I think your only option would be to rename your LVM devices to not include hyphens. Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1553 Mon, 09 Apr 2012 15:51:38 +0000 Anonymous 1553@http://www.datastax.com/support-forums/ Sure : The output of "df --print-type --local" is : Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/cnode03-root ext4 667630200 2763032 630953520 1% / udev devtmpfs 18564420 4 18564416 1% /dev tmpfs tmpfs 7429388 316 7429072 1% /run none tmpfs 5120 0 5120 0% /run/lock none tmpfs 18573464 0 18573464 0% /run/shm /dev/sda1 ext2 233191 46057 174693 21% /boot /dev/sdb1 ext4 140998528 825368 133010844 1% /commitlog And the output of "curl <opscenterd_ip>:8888/nodes/all/partitions" (10.10.107.2 is my OpsCenter server) is : {"10.10.107.1": {"saved_caches": "/dev/mapper/cnode01_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode01_root"]}, "10.10.107.2": {"saved_caches": "/dev/mapper/cnode03_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode03_root"]}, "10.10.117.2": {"saved_caches": "/dev/mapper/cnode04_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode04_root"]}, "10.10.117.1": {"saved_caches": "/dev/mapper/cnode02_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode02_root"]}} thobbs on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1549 Fri, 06 Apr 2012 19:07:54 +0000 thobbs 1549@http://www.datastax.com/support-forums/ On node 10.10.107.2, can I get the output of "<code>df --print-type --local</code>"? Can I also get the output of "<code>curl <opscenterd_ip>:8888/nodes/all/partitions</code>"? Anonymous on "OpsCenter Rebalance cluster" http://www.datastax.com/support-forums/topic/opscenter-rebalance-cluster#post-1543 Fri, 06 Apr 2012 12:52:36 +0000 Anonymous 1543@http://www.datastax.com/support-forums/ Hello, I've installed DSE 2.0 in a Ubuntu 11.10 server through apt-get by following Datastax installation procedure. Everythink work well except the "Cluster rebalance" function in OpsCenter. I have currently two node, and here is the opscenter.log content where I try to rebalance : 2012-04-06 14:24:20+0200 [] INFO: Starting rebalance 2012-04-06 14:24:20+0200 [] INFO: Reblance: Moving node 10.10.107.2 from token 143594533192801367272885573762123582805 to 105217455362211895469402111707997394652 2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last): File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 388, in errback self._startRunCallbacks(fail) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks self._runCallbacks() File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1076, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/ClusterUtils.py", line 347, in _runNext File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/cluster/Rebalance.py", line 141, in move File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/cluster/Rebalance.py", line 121, in _doMove File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 185, in move File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 300, in checkMoveDataSizes File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 225, in sumDiskSpace exceptions.KeyError: u'/dev/mapper/cnode03_root' 2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last): File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 388, in errback self._startRunCallbacks(fail) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks self._runCallbacks() File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1076, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/opscenterd/ClusterUtils.py", line 336, in run exceptions.KeyError: u'/dev/mapper/cnode03_root' 2012-04-06 14:24:20+0200 [] ERROR: Rebalance failed: u'/dev/mapper/cnode03_root' 2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last): Failure: twisted.python.failure.DefaultException: u'/dev/mapper/cnode03_root'