Hey guys,
we're in the process of evaluating the datastax platforms, and have a working brisk cluster, however oddly we're not seeing any disk space utilization on the consoles, and we're also noticing some errors in the agents log.
Attached is an excerpt from our logs:
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Received SIGTERM, shutting down.
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Unhandled error in Deferred:
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Unhandled Error
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Traceback (most recent call last):
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/base.py", line 407, in fireEvent
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = callable(*args, **kwargs)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/application/service.py", line 285, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] l.append(defer.maybeDeferred(service.stopService))
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = f(*args, **kw)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/application/service.py", line 285, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] l.append(defer.maybeDeferred(service.stopService))
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] --- <exception caught here> ---
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = f(*args, **kw)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/usr/lib/python2.6/site-packages/opsagent/StatsCollection.py", line 34, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-]
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/task.py", line 171, in stop
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] assert self.running, ("Tried to stop a LoopingCall that was "
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] exceptions.AssertionError: Tried to stop a LoopingCall that was not running.
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Unhandled error in Deferred:
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Unhandled Error
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Traceback (most recent call last):
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/base.py", line 407, in fireEvent
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = callable(*args, **kwargs)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/application/service.py", line 285, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] l.append(defer.maybeDeferred(service.stopService))
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = f(*args, **kw)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/application/service.py", line 285, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] l.append(defer.maybeDeferred(service.stopService))
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] --- <exception caught here> ---
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] result = f(*args, **kw)
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/usr/lib/python2.6/site-packages/opsagent/MetricService.py", line 54, in stopService
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-]
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/task.py", line 171, in stop
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] assert self.running, ("Tried to stop a LoopingCall that was "
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] exceptions.AssertionError: Tried to stop a LoopingCall that was not running.
Jun 1 22:46:24 aklcas02 'opscenter-agent': [StompProtocol,client] <twisted.internet.ssl.Connector instance at 0x1171a2d8> will retry in 1 seconds
Jun 1 22:46:24 aklcas02 'opscenter-agent': [StompProtocol,client] Stopping factory <opsagent.OpsAgentService.StompClientFactory instance at 0x116a7128>
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Main loop terminated.
Jun 1 22:46:24 aklcas02 'opscenter-agent': [-] Server Shut Down.
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] Log opened.
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] using set_wakeup_fd
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up.
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] reactor class: twisted.internet.selectreactor.SelectReactor.
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] Starting factory <opsagent.OpsAgentService.StompClientFactory instance at 0xd01c170>
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] Unhandled error in Deferred:
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] Unhandled Error
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] Traceback (most recent call last):
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 361, in callback
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] self._startRunCallbacks(result)
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 455, in _startRunCallbacks
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] self._runCallbacks()
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 542, in _runCallbacks
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] current.result = callback(current.result, *args, **kw)
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1076, in gotResult
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] _inlineCallbacks(r, g, deferred)
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] --- <exception caught here> ---
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/opt/opscenter/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1020, in _inlineCallbacks
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] result = g.send(result)
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] File "/usr/lib/python2.6/site-packages/opsagent/StatsCollection.py", line 208, in _collect_disk_space_stats
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-]
Jun 1 22:46:27 aklcas02 'opscenter-agent': [-] exceptions.IndexError: list index out of range
