OpsCenter Rebalance cluster

(11 posts) (3 voices)

• Started 1 year ago by
• Latest reply from nickmbailey
• This topic is resolved

Tags:

• winter formal dresses 2011

1. 

   Anonymous
   Unregistered

   Hello,

   I've installed DSE 2.0 in a Ubuntu 11.10 server through apt-get by following Datastax installation procedure.

   Everythink work well except the "Cluster rebalance" function in OpsCenter.

   I have currently two node, and here is the opscenter.log content where I try to rebalance :

   2012-04-06 14:24:20+0200 [] INFO: Starting rebalance
   2012-04-06 14:24:20+0200 [] INFO: Reblance: Moving node 10.10.107.2 from token 143594533192801367272885573762123582805 to 105217455362211895469402111707997394652
   2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last):
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 388, in errback
   self._startRunCallbacks(fail)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks
   self._runCallbacks()
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks
   current.result = callback(current.result, *args, **kw)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1076, in gotResult
   _inlineCallbacks(r, g, deferred)
   --- <exception caught here> ---
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/ClusterUtils.py", line 347, in _runNext

   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/cluster/Rebalance.py", line 141, in move

   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/cluster/Rebalance.py", line 121, in _doMove

   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 185, in move

   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 300, in checkMoveDataSizes

   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks
   current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python2.7/dist-packages/opscenterd/Cluster.py", line 225, in sumDiskSpace

   exceptions.KeyError: u'/dev/mapper/cnode03_root'

   2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last):
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 388, in errback
   self._startRunCallbacks(fail)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks
   self._runCallbacks()
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks
   current.result = callback(current.result, *args, **kw)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1076, in gotResult
   _inlineCallbacks(r, g, deferred)
   --- <exception caught here> ---
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
   result = result.throwExceptionIntoGenerator(g)
   File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
   return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python2.7/dist-packages/opscenterd/ClusterUtils.py", line 336, in run

   exceptions.KeyError: u'/dev/mapper/cnode03_root'

   2012-04-06 14:24:20+0200 [] ERROR: Rebalance failed: u'/dev/mapper/cnode03_root'
   2012-04-06 14:24:20+0200 [] ERROR: Traceback (most recent call last):
   Failure: twisted.python.failure.DefaultException: u'/dev/mapper/cnode03_root'

   Posted 1 year ago #
2. 

   thobbs
   Administrator

   On node 10.10.107.2, can I get the output of "df --print-type --local"?

   Can I also get the output of "curl <opscenterd_ip>:8888/nodes/all/partitions"?

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   Sure :

   The output of "df --print-type --local" is :

   Filesystem Type 1K-blocks Used Available Use% Mounted on
   /dev/mapper/cnode03-root
   ext4 667630200 2763032 630953520 1% /
   udev devtmpfs 18564420 4 18564416 1% /dev
   tmpfs tmpfs 7429388 316 7429072 1% /run
   none tmpfs 5120 0 5120 0% /run/lock
   none tmpfs 18573464 0 18573464 0% /run/shm
   /dev/sda1 ext2 233191 46057 174693 21% /boot
   /dev/sdb1 ext4 140998528 825368 133010844 1% /commitlog

   And the output of "curl <opscenterd_ip>:8888/nodes/all/partitions" (10.10.107.2 is my OpsCenter server) is :

   {"10.10.107.1": {"saved_caches": "/dev/mapper/cnode01_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode01_root"]}, "10.10.107.2": {"saved_caches": "/dev/mapper/cnode03_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode03_root"]}, "10.10.117.2": {"saved_caches": "/dev/mapper/cnode04_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode04_root"]}, "10.10.117.1": {"saved_caches": "/dev/mapper/cnode02_root", "commitlog": "/dev/sdb1", "other": ["/dev/sda1"], "data": ["/dev/mapper/cnode02_root"]}}

   Posted 1 year ago #
4. 

   thobbs
   Administrator

   Thanks for the output, I can definitely see what's broken here: there's a minor mismatch in how device names are escaped.

   I've already fixed this for the next release of OpsCenter. Unfortunately, there isn't really a good workaround for this in 1.4.1. I think your only option would be to rename your LVM devices to not include hyphens.

   Posted 1 year ago #
5. 

   Anonymous
   Unregistered

   Many thank's Thobbs, I try the workaround and keep you informed.
   Do you know when the next release (with fix) is planned ?

   Posted 1 year ago #
6. 

   thobbs
   Administrator

   It's hard to give exact dates, but hopefully we will have a release out that includes the fix within 2 to 3 weeks.

   Posted 1 year ago #
7. 

   Anonymous
   Unregistered

   Unfortunately, the device name /dev/mapper/cnode03-root seems to be managed directly manager by LVM 2 (this is the default configuration for Ubuntu server 11.10) the mapping does not appear to be changed.

   Unless you have a procedure to do this, we will have to wait for the OpsCenter patch (or include it on the current version).

   Posted 1 year ago #
8. 

   thobbs
   Administrator

   Ah, sorry about that. It looks like we picked the 17th as the date for the OpsCenter release that will include the fix. Try out the new version when it's released and let us know if everything is resolved.

   Posted 1 year ago #
9. 

   Anonymous
   Unregistered

   Ok, we wait for this and I keep you informed.

   Posted 1 year ago #
10. 

    Anonymous
    Unregistered

    Hello Thobbs,

    Just to configm you the patched version released today work fine.
    Our rebalance cluster problem is fixed.
    Thank for this good work.

    The case can be closed.

    Posted 1 year ago #
11. 

    nickmbailey
    Key Master

    Glad to hear that the issue is resolved! Let us know if you have any other issues or feedback.

    Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.