Error Connecting to Cluster

(11 posts) (3 voices)

• Started 1 year ago by geoff.wagner
• Latest reply from geoff.wagner
• This topic is resolved

Tags:

No tags yet.

1. 

   geoff.wagner
   Member

   I am having trouble connecting to my 2 box cassandra test cluster. The install of the package went fine, but now i can not connect to the cluster.

   I have tried connecting to the cluster by installing Opscenter on a box with a cassandra node (using 127.0.0.1 and the box's ip address), and from a remote node.

   When i try and up my log level to trace i dont get any extra information.

   Here is the log for Opscenter:

   2011-08-11 10:35:00-0500 [] INFO: Log opened.
   2011-08-11 10:35:00-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up.
   2011-08-11 10:35:00-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
   2011-08-11 10:35:00-0500 [] INFO: set uid/gid 101/103
   2011-08-11 10:35:00-0500 [] INFO: Logging level set to 'trace'
   2011-08-11 10:35:00-0500 [] INFO: OpsCenterdService startService
   2011-08-11 10:35:00-0500 [] INFO: OpsCenter version: 1.2.2
   2011-08-11 10:35:00-0500 [] INFO: Compatible agent version: 2.2
   2011-08-11 10:35:00-0500 [] DEBUG: Logging user config options:
   2011-08-11 10:35:00-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'TRACE')], webserver : [('interface', '10.45.8.64'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')],
   2011-08-11 10:35:00-0500 [] INFO: HTTP BASIC authentication disabled
   2011-08-11 10:35:00-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199}
   2011-08-11 10:35:00-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
   2011-08-11 10:35:00-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x995698c>
   2011-08-11 10:35:00-0500 [] INFO: Unhandled error in Deferred:
   2011-08-11 10:35:00-0500 [] INFO:
   2011-08-11 10:35:04-0500 [] INFO:
   2011-08-11 10:35:10-0500 [] INFO:
   2011-08-11 10:39:11-0500 [] INFO:
   2011-08-11 10:39:35-0500 [] INFO:
   2011-08-11 10:40:02-0500 [] INFO: Received SIGTERM, shutting down.
   2011-08-11 10:40:02-0500 [] INFO: Stopping CassandraCluster service
   2011-08-11 10:40:02-0500 [] INFO: JmxJythonService is now in state "dying"
   2011-08-11 10:40:02-0500 [] INFO: (TCP Port 8888 Closed)
   2011-08-11 10:40:02-0500 [] INFO: Stopping factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x995698c>
   2011-08-11 10:40:02-0500 [] INFO: Main loop terminated.
   2011-08-11 10:40:02-0500 [] INFO: Server Shut Down.
   2011-08-11 10:40:02-0500 [] INFO: Log opened.
   2011-08-11 10:40:02-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up.
   2011-08-11 10:40:02-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
   2011-08-11 10:40:02-0500 [] INFO: set uid/gid 101/103
   2011-08-11 10:40:02-0500 [] INFO: Logging level set to 'debug'
   2011-08-11 10:40:02-0500 [] INFO: OpsCenterdService startService
   2011-08-11 10:40:02-0500 [] INFO: OpsCenter version: 1.2.2
   2011-08-11 10:40:02-0500 [] INFO: Compatible agent version: 2.2
   2011-08-11 10:40:02-0500 [] DEBUG: Logging user config options:
   2011-08-11 10:40:02-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'DEBUG')], webserver : [('interface', '10.45.8.64'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')],
   2011-08-11 10:40:02-0500 [] INFO: HTTP BASIC authentication disabled
   2011-08-11 10:40:02-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199}
   2011-08-11 10:40:02-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
   2011-08-11 10:40:02-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0xa06298c>
   2011-08-11 10:40:02-0500 [] INFO: Unhandled error in Deferred:
   2011-08-11 10:40:02-0500 [] INFO:
   2011-08-11 10:40:06-0500 [] INFO:

   Also, when I turn up the logging on cassandra, i dont see any hit from my Opscenter box. I am currently running 'Sébastien Giroux' admin tool and it can talk to the cluster with no problems from any machine. There is NO firewall between any of these servers.

   Posted 1 year ago #
2. 

   nickmbailey
   Key Master

   At first glance it looks like a problem with the jython process that is spawned in order to communicate through jmx.

   Can run the following command and see what the output/errors are?

   CLASSPATH="/usr/share/opscenter/bin/jmxjython.jar:/usr/share/opscenter/lib/*" /usr/bin/java -Djava.library.path=/usr/lib/jni -Dpython.cachedir=/usr/share/opscenter/.jython_cache org.python.util.jython -jar /usr/share/opscenter/bin/jmxjython.jar -t 50 -k 60 --automated

   Posted 1 year ago #
3. 

   geoff.wagner
   Member

   I ran that on both the Opscenter server and my Cassandra server and the output they both spit out

   Y(I-1
   S'YAY\n'
   p0
   tp1
   .?

   and then they hang and they dont return to the command prompt

   Posted 1 year ago #
4. 

   nickmbailey
   Key Master

   That is actually the expected result. It appears this problem may result when attempting to bind the webserver to an interface the machine doesn't know about. What interface are you attempting to bind the webserver to? Can you see if binding to 127.0.0.1 or 0.0.0.0 works?

   Posted 1 year ago #
5. 

   geoff.wagner
   Member

   That did not work. Same error. This URL is to a screenshot: http://postimage.org/image/12er21eqs/

   Btw i am on CentOS 5.5 installed version 1.2.2

   Posted 1 year ago #
6. 

   nickmbailey
   Key Master

   There is a currently a bug with logging that is causing the exception in your logs to disappear which makes this slightly harder to debug. It sounds like OpsCenter simply can't talk to the cluster over thrift however. Possible causes:

   1. Did you change the default thrift port?
   2. Do you have cassandra authentication enabled?
   3. Is there a firewall blocking OpsCenter from talking to node 10.45.8.61 on the thrift port?

   Posted 1 year ago #
7. 

   geoff.wagner
   Member

   No to all three. I have a different application on the same server as opscenter that has no problem connecting to Cassandra. Can I get a new rpm with the logging bug fixed to help narrow down the problem?

   Posted 1 year ago #
8. 

   mbulman
   Key Master

   A patched version is available (1.2.3). Just run yum clean all && yum install opscenter.

   Posted 1 year ago #
9. 

   geoff.wagner
   Member

   Thanks for the Fix!! Here is a new startup log:

   2011-08-15 08:23:10-0500 [] INFO: Log opened.
   2011-08-15 08:23:10-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up.
   2011-08-15 08:23:10-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
   2011-08-15 08:23:10-0500 [] INFO: set uid/gid 101/103
   2011-08-15 08:23:10-0500 [] INFO: Logging level set to 'debug'
   2011-08-15 08:23:10-0500 [] INFO: OpsCenterdService startService
   2011-08-15 08:23:10-0500 [] INFO: OpsCenter version: 1.2.3
   2011-08-15 08:23:10-0500 [] INFO: Compatible agent version: 2.2
   2011-08-15 08:23:10-0500 [] DEBUG: Logging user config options:
   2011-08-15 08:23:10-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'DEBUG')], webserver : [('interface', '0.0.0.0'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')],
   2011-08-15 08:23:10-0500 [] INFO: HTTP BASIC authentication disabled
   2011-08-15 08:23:10-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199}
   2011-08-15 08:23:10-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
   2011-08-15 08:23:10-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x92359ec>
   2011-08-15 08:23:10-0500 [] INFO: Unhandled error in Deferred:
   2011-08-15 08:23:10-0500 [] Unhandled Error
   Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/twisted/scripts/_twistd_unix.py", line 317, in startApplication
   app.startApplication(application, not self.config['no_save'])
   File "/usr/lib/python2.6/site-packages/twisted/application/app.py", line 653, in startApplication
   service.IService(application).startService()
   File "/usr/lib/python2.6/site-packages/twisted/application/service.py", line 277, in startService
   service.startService()
   File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1141, in unwindGenerator
   return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
   --- <exception caught here> ---
   File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1020, in _inlineCallbacks
   result = g.send(result)
   File "/usr/lib/python2.6/site-packages/opscenterd/OpsCenterdService.py", line 223, in startService

   File "/usr/lib/python2.6/site-packages/twisted/application/service.py", line 277, in startService
   service.startService()
   File "/usr/lib/python2.6/site-packages/opscenterd/jmxadapt/JmxJythonService.py", line 318, in startService

   File "/usr/lib/python2.6/site-packages/opscenterd/jmxadapt/JmxJythonService.py", line 346, in setupJythonEnv

   File "/usr/lib/python2.6/os.py", line 157, in makedirs
   mkdir(name, mode)
   exceptions.OSError: [Errno 13] Permission denied: '/root/.jython_cache'

   Looks like a mkdir problem to me. My root directory permissions looks like this:

   drwxr-x--- 8 root root 4096 Aug 15 08:23 root

   Also i am starting the service as root (which i understand there is an opscenter user setup). Must be a user var somewhere reading the ENV_VAR for the current running user in once place, and expecting your predefined opscenter user in a different place. I am guessing in your code, you do a change over to run as the opscenter user, and the opscenter user can not access /root/.jython_cache.

   Posted 1 year ago #
10. 

    mbulman
    Key Master

    What command are you using to start the opscenterd service? We've seen some weirdness in the past when using /etc/init.d/opscenterd directly. We recommend using service opscenterd start|stop|restart to make sure things behave the way we expect. Is that the case here?

    Posted 1 year ago #
11. 

    geoff.wagner
    Member

    /sbin/service opscenterd start worked perfectly

    Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.