DataStax Support Forums » Topic: Error Connecting to Cluster

DataStax Support Forums » Topic: Error Connecting to Cluster http://www.datastax.com/support-forums/topic/error-connecting-to-cluster Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 03:46:03 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-441 Mon, 15 Aug 2011 15:58:20 +0000 geoff.wagner 441@http://www.datastax.com/support-forums/ /sbin/service opscenterd start worked perfectly mbulman on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-440 Mon, 15 Aug 2011 15:41:54 +0000 mbulman 440@http://www.datastax.com/support-forums/ What command are you using to start the opscenterd service? We've seen some weirdness in the past when using /etc/init.d/opscenterd directly. We recommend using <code>service opscenterd start|stop|restart</code> to make sure things behave the way we expect. Is that the case here? geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-439 Mon, 15 Aug 2011 13:25:16 +0000 geoff.wagner 439@http://www.datastax.com/support-forums/ Thanks for the Fix!! Here is a new startup log: <blockquote> 2011-08-15 08:23:10-0500 [] INFO: Log opened. 2011-08-15 08:23:10-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up. 2011-08-15 08:23:10-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor. 2011-08-15 08:23:10-0500 [] INFO: set uid/gid 101/103 2011-08-15 08:23:10-0500 [] INFO: Logging level set to 'debug' 2011-08-15 08:23:10-0500 [] INFO: OpsCenterdService startService 2011-08-15 08:23:10-0500 [] INFO: OpsCenter version: 1.2.3 2011-08-15 08:23:10-0500 [] INFO: Compatible agent version: 2.2 2011-08-15 08:23:10-0500 [] DEBUG: Logging user config options: 2011-08-15 08:23:10-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'DEBUG')], webserver : [('interface', '0.0.0.0'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')], 2011-08-15 08:23:10-0500 [] INFO: HTTP BASIC authentication disabled 2011-08-15 08:23:10-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199} 2011-08-15 08:23:10-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-08-15 08:23:10-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x92359ec> 2011-08-15 08:23:10-0500 [] INFO: Unhandled error in Deferred: 2011-08-15 08:23:10-0500 [] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/twisted/scripts/_twistd_unix.py", line 317, in startApplication app.startApplication(application, not self.config['no_save']) File "/usr/lib/python2.6/site-packages/twisted/application/app.py", line 653, in startApplication service.IService(application).startService() File "/usr/lib/python2.6/site-packages/twisted/application/service.py", line 277, in startService service.startService() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1141, in unwindGenerator return _inlineCallbacks(None, f(*args, **kwargs), Deferred()) --- <exception caught here> --- File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 1020, in _inlineCallbacks result = g.send(result) File "/usr/lib/python2.6/site-packages/opscenterd/OpsCenterdService.py", line 223, in startService File "/usr/lib/python2.6/site-packages/twisted/application/service.py", line 277, in startService service.startService() File "/usr/lib/python2.6/site-packages/opscenterd/jmxadapt/JmxJythonService.py", line 318, in startService File "/usr/lib/python2.6/site-packages/opscenterd/jmxadapt/JmxJythonService.py", line 346, in setupJythonEnv File "/usr/lib/python2.6/os.py", line 157, in makedirs mkdir(name, mode) exceptions.OSError: [Errno 13] Permission denied: '/root/.jython_cache' </blockquote> Looks like a mkdir problem to me. My root directory permissions looks like this: <blockquote> drwxr-x--- 8 root root 4096 Aug 15 08:23 root </blockquote> Also i am starting the service as root (which i understand there is an opscenter user setup). Must be a user var somewhere reading the ENV_VAR for the current running user in once place, and expecting your predefined opscenter user in a different place. I am guessing in your code, you do a change over to run as the opscenter user, and the opscenter user can not access /root/.jython_cache. mbulman on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-437 Sat, 13 Aug 2011 23:02:49 +0000 mbulman 437@http://www.datastax.com/support-forums/ A patched version is available (1.2.3). Just run <code>yum clean all && yum install opscenter</code>. geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-436 Sat, 13 Aug 2011 21:23:23 +0000 geoff.wagner 436@http://www.datastax.com/support-forums/ No to all three. I have a different application on the same server as opscenter that has no problem connecting to Cassandra. Can I get a new rpm with the logging bug fixed to help narrow down the problem? nickmbailey on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-435 Fri, 12 Aug 2011 22:41:09 +0000 nickmbailey 435@http://www.datastax.com/support-forums/ There is a currently a bug with logging that is causing the exception in your logs to disappear which makes this slightly harder to debug. It sounds like OpsCenter simply can't talk to the cluster over thrift however. Possible causes: 1. Did you change the default thrift port? 2. Do you have cassandra authentication enabled? 3. Is there a firewall blocking OpsCenter from talking to node 10.45.8.61 on the thrift port? geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-431 Fri, 12 Aug 2011 00:18:19 +0000 geoff.wagner 431@http://www.datastax.com/support-forums/ That did not work. Same error. This URL is to a screenshot: <a href="http://postimage.org/image/12er21eqs/" rel="nofollow">http://postimage.org/image/12er21eqs/</a> Btw i am on CentOS 5.5 installed version 1.2.2 nickmbailey on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-424 Thu, 11 Aug 2011 20:58:40 +0000 nickmbailey 424@http://www.datastax.com/support-forums/ That is actually the expected result. It appears this problem may result when attempting to bind the webserver to an interface the machine doesn't know about. What interface are you attempting to bind the webserver to? Can you see if binding to 127.0.0.1 or 0.0.0.0 works? geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-422 Thu, 11 Aug 2011 17:48:50 +0000 geoff.wagner 422@http://www.datastax.com/support-forums/ I ran that on both the Opscenter server and my Cassandra server and the output they both spit out <blockquote> Y(I-1 S'YAY\n' p0 tp1 .? </blockquote> and then they hang and they dont return to the command prompt nickmbailey on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-421 Thu, 11 Aug 2011 16:17:26 +0000 nickmbailey 421@http://www.datastax.com/support-forums/ At first glance it looks like a problem with the jython process that is spawned in order to communicate through jmx. Can run the following command and see what the output/errors are? <blockquote> <code>CLASSPATH="/usr/share/opscenter/bin/jmxjython.jar:/usr/share/opscenter/lib/*" /usr/bin/java -Djava.library.path=/usr/lib/jni -Dpython.cachedir=/usr/share/opscenter/.jython_cache org.python.util.jython -jar /usr/share/opscenter/bin/jmxjython.jar -t 50 -k 60 --automated</code> </blockquote> geoff.wagner on "Error Connecting to Cluster" http://www.datastax.com/support-forums/topic/error-connecting-to-cluster#post-420 Thu, 11 Aug 2011 15:46:57 +0000 geoff.wagner 420@http://www.datastax.com/support-forums/ I am having trouble connecting to my 2 box cassandra test cluster. The install of the package went fine, but now i can not connect to the cluster. I have tried connecting to the cluster by installing Opscenter on a box with a cassandra node (using 127.0.0.1 and the box's ip address), and from a remote node. When i try and up my log level to trace i dont get any extra information. Here is the log for Opscenter: <blockquote> 2011-08-11 10:35:00-0500 [] INFO: Log opened. 2011-08-11 10:35:00-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up. 2011-08-11 10:35:00-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor. 2011-08-11 10:35:00-0500 [] INFO: set uid/gid 101/103 2011-08-11 10:35:00-0500 [] INFO: Logging level set to 'trace' 2011-08-11 10:35:00-0500 [] INFO: OpsCenterdService startService 2011-08-11 10:35:00-0500 [] INFO: OpsCenter version: 1.2.2 2011-08-11 10:35:00-0500 [] INFO: Compatible agent version: 2.2 2011-08-11 10:35:00-0500 [] DEBUG: Logging user config options: 2011-08-11 10:35:00-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'TRACE')], webserver : [('interface', '10.45.8.64'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')], 2011-08-11 10:35:00-0500 [] INFO: HTTP BASIC authentication disabled 2011-08-11 10:35:00-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199} 2011-08-11 10:35:00-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-08-11 10:35:00-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x995698c> 2011-08-11 10:35:00-0500 [] INFO: Unhandled error in Deferred: 2011-08-11 10:35:00-0500 [] INFO: 2011-08-11 10:35:04-0500 [] INFO: 2011-08-11 10:35:10-0500 [] INFO: 2011-08-11 10:39:11-0500 [] INFO: 2011-08-11 10:39:35-0500 [] INFO: 2011-08-11 10:40:02-0500 [] INFO: Received SIGTERM, shutting down. 2011-08-11 10:40:02-0500 [] INFO: Stopping CassandraCluster service 2011-08-11 10:40:02-0500 [] INFO: JmxJythonService is now in state "dying" 2011-08-11 10:40:02-0500 [] INFO: (TCP Port 8888 Closed) 2011-08-11 10:40:02-0500 [] INFO: Stopping factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x995698c> 2011-08-11 10:40:02-0500 [] INFO: Main loop terminated. 2011-08-11 10:40:02-0500 [] INFO: Server Shut Down. 2011-08-11 10:40:02-0500 [] INFO: Log opened. 2011-08-11 10:40:02-0500 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.5) starting up. 2011-08-11 10:40:02-0500 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor. 2011-08-11 10:40:02-0500 [] INFO: set uid/gid 101/103 2011-08-11 10:40:02-0500 [] INFO: Logging level set to 'debug' 2011-08-11 10:40:02-0500 [] INFO: OpsCenterdService startService 2011-08-11 10:40:02-0500 [] INFO: OpsCenter version: 1.2.2 2011-08-11 10:40:02-0500 [] INFO: Compatible agent version: 2.2 2011-08-11 10:40:02-0500 [] DEBUG: Logging user config options: 2011-08-11 10:40:02-0500 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'DEBUG')], webserver : [('interface', '10.45.8.64'), ('port', '8888')], authentication : [], cassandra_metrics : [('ignored_keyspaces', 'system, OpsCenter')], cassandra : [('seed_hosts', '10.45.8.61')], 2011-08-11 10:40:02-0500 [] INFO: HTTP BASIC authentication disabled 2011-08-11 10:40:02-0500 [] INFO: agent_config items: {'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199} 2011-08-11 10:40:02-0500 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-08-11 10:40:02-0500 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0xa06298c> 2011-08-11 10:40:02-0500 [] INFO: Unhandled error in Deferred: 2011-08-11 10:40:02-0500 [] INFO: 2011-08-11 10:40:06-0500 [] INFO: </blockquote> Also, when I turn up the logging on cassandra, i dont see any hit from my Opscenter box. I am currently running 'Sébastien Giroux' admin tool and it can talk to the cluster with no problems from any machine. There is NO firewall between any of these servers.