Support Forums : DataStax

DataStax Support Forums » OpsCenter

Opscenter Agent BindExceptions

(12 posts) (5 voices)

Started 1 year ago by
Latest reply from mbulman
This topic is not a support question

Tags:

Anonymous
Unregistered

I think this may have been covered in another post but wanted to report it and see what the approximate timeline would be for a fix making it into OpsCenter Community.

I have a Cassandra cluster with 2 datacenters. 1 on aws and 1 private. I have opscenter running on a seperate instance on AWS.

Opscenter connects to the cluster fine and finds all the nodes, pushed out the agents no problem. Agents launch. I see all nodes as active in the OpsCenter dashboard, I see the different datacenters and stats and I see all nodes connected in the storage pie chart as well.

All good. Now if I go to the cluster and actually click on a node and choose "view replication" for example I get an error that connection was refused. When I check the logs for the agent on that machine I see Jetty BindException cannot assign address.

It seems that Jetty tries to bind to the public IP of the node as listed in the address.yaml pushed out by OpsCenter.

So if I change the local address in the address.yaml to be the private address of the node, OR to be 0.0.0.0 then I no longer get jetty errors in the logs and the actions in opscenter work (ie view replication now works). But the node appears disconnected in the storage pie graph on the front page and opscenter starts prompting me to install the agent again on that node.

So it seems like there needs to be a way to tell Jetty to bind to one address but still use the proper "local" address so that opscenter knows what address to contact the node on.

Is there anyway I can over-ride this for the time being either editing startup scripts or similar to force jetty to just bind 0.0.0.0 and still keep opscenter happy by having the node local address that it needs in the address.yaml?

Posted 1 year ago #
mbulman
Key Master

Hi,

We're working on getting a fix for this in the near future. I don't have an exact timeframe at this point, and unfortunately there is no workaround.

Posted 1 year ago #
Anonymous
Unregistered

Is there a JIRA/BUG number that we could reference for tracking purposes? I am also facing this issue.

Posted 1 year ago #
nickmbailey
Key Master

Unfortunately there is no public bug tracker for OpsCenter. We do have this forum post noted on the specific bug ticket. We will update this forum post when we release a version that resolves this issue.

We try and make sure we do this for all bugs reported on the forums.

Posted 1 year ago #
nickmbailey
Key Master

I'm happy to report that this issue should have been resolved with the very recent 2.0 release of OpsCenter. In this release you can change the interface the agent binds to by specifying 'agent_rpc_interface' in the conf for the agent. In package installs this file will be locatedated at /var/lib/opscenter-agent/conf/address.yaml. In manual installs it will be located at <install_dir>/conf/address.yaml.

Let us know if this fix works for you.

Posted 1 year ago #

Anonymous
Unregistered

I'm not sure if it's related, but I've been trying to get the opscenter agents upgraded from 1.4 to 2.0 and they keep throwing Java exceptions:

INFO [main] 2012-04-19 13:27:43,385 Loading conf files: /usr/share/opscenter-agent/opscenter-agent-2.7-standalone.jar, /var/lib/opscenter-agent/conf/address.yaml
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Character cannot be cast to java.util.Map$Entry
	at clojure.lang.LazySeq.sval(LazySeq.java:47)
	at clojure.lang.LazySeq.seq(LazySeq.java:56)
	at clojure.lang.RT.seq(RT.java:450)
	at clojure.core$seq.invoke(core.clj:122)
	at clojure.core$r.invoke(core.clj:793)
	at clojure.core$into.invoke(core.clj:2614)
	at opsagent.conf$load_conf.invoke(conf.clj:89)
	at opsagent.opsagent$_main.doInvoke(opsagent.clj:169)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at opsagent.opsagent.main(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Character cannot be cast to java.util.Map$Entry
	at clojure.lang.LazySeq.sval(LazySeq.java:47)
	at clojure.lang.LazySeq.seq(LazySeq.java:56)
	at clojure.lang.RT.seq(RT.java:450)
	at clojure.core$seq.invoke(core.clj:122)
	at clojure.core$r.invoke(core.clj:793)
	at clojure.core$into.invoke(core.clj:2614)
	at opsagent.conf$load_conf_file.invoke(conf.clj:85)
	at clojure.core$map$fn__425.invoke(core.clj:2096)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	... 9 more
Caused by: java.lang.ClassCastException: java.lang.Character cannot be cast to java.util.Map$Entry
	at opsagent.conf$load_conf_file$fn__313.invoke(conf.clj:84)
	at clojure.core$map$fn__425.invoke(core.clj:2096)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	... 17 more

My /var/lib/opscenter-agent/conf/address.yaml file only contains these two lines:

stomp_interface: "<ip of opscenterd>"
agent_rpc_interface: "<ip of local machine>"

with obviously the IP addresses filled out.

any ideas? This is on a CentOS 5 machine. Thanks.

Posted 1 year ago #

nickmbailey
Key Master

That is indicating that the agent is trying to load a jar file as a configuration file. The old agent jar file did not get cleared out of '/usr/share/opscenter-agent/'. If you delete the jar file in that directory with the older version number it should resolve the problem.

How did you install the agents in this case? Using the OpsCenter UI? Manually with the agent rpms? Manually with the agent tarball?

Posted 1 year ago #

Anonymous
Unregistered

So what should be in /usr/share/opscenter-agent? Right now I'm seeing:

[root@test opscenter-agent]# pwd
/usr/share/opscenter-agent
[root@test opscenter-agent]# ll
total 41108
drwxr-xr-x 2 opscenter-agent opscenter-agent     4096 Apr 19 12:56 bin
drwxr-xr-x 2 opscenter       opscenter           4096 Apr 19 11:28 conf
-rwxr-xr-x 1 opscenter       opscenter          18563 Apr 17 11:48 LICENSE
-rwxr-xr-x 1 opscenter       opscenter       11569316 Feb 24 13:12 opscenter-agent-2.6-standalone.jar
-rwxr-xr-x 1 opscenter-agent opscenter-agent 15864771 Apr 17 11:53 opscenter-agent-2.7-standalone.jar
-rwxr-xr-x 1 opscenter       opscenter       14567887 Apr 17 11:53 opscenter-agent.rpm

I installed it like:

cd /usr/share
tar zxf opscenter-agent.tar.gz
cd opscenter-agent
./bin/install_agent.sh opscenter-agent.rpm <opscenterd_host>

I may have had an earlier version installed in the same location though, which might have been the problem.

Posted 1 year ago #

nickmbailey
Key Master

Yes I believe that caused the problem. The jar file from the old installation is still there. You need to delete 'opscenter-agent-2.6-standalone.jar'

'rm /usr/share/opscenter-agent/opscenter-agent-2.6-standalone.jar'

Posted 1 year ago #
Anonymous
Unregistered

That did it. Thanks!

Posted 1 year ago #

Anonymous
Unregistered

Related to this, it looks like opscenter-agent-2.6-standalone.jar is coming in the opscenter-agent.tar.gz file that is suppose to be installed on all of the nodes:

[root@server share]# tar tvzf opscenter-agent.tar.gz
drwxr-xr-x opscenter/opscenter 0 2012-04-19 11:28 opscenter-agent/
-rwxr-xr-x opscenter/opscenter 11569316 2012-02-24 13:12 opscenter-agent/opscenter-agent-2.6-standalone.jar
drwxr-xr-x opscenter/opscenter        0 2012-04-19 11:28 opscenter-agent/conf/
-rwxr-xr-x opscenter/opscenter      684 2012-04-17 11:27 opscenter-agent/conf/log4j.properties
-rwxr-xr-x opscenter/opscenter 14567887 2012-04-17 11:53 opscenter-agent/opscenter-agent.rpm
-rwxr-xr-x opscenter/opscenter 15864785 2012-04-17 12:00 opscenter-agent/opscenter-agent-2.7-standalone.jar
drwxr-xr-x opscenter/opscenter        0 2012-04-19 11:28 opscenter-agent/bin/
-rwxr-xr-x opscenter/opscenter     2198 2012-04-17 11:27 opscenter-agent/bin/opscenter-agent
-rwxr-xr-x opscenter/opscenter     2127 2012-04-17 11:27 opscenter-agent/bin/install_agent.sh
-rw-r--r-- root/root               1925 2012-04-19 11:28 opscenter-agent/bin/agentKeyStore
-rwxr-xr-x opscenter/opscenter      377 2012-04-17 11:27 opscenter-agent/bin/setup
-rwxr-xr-x opscenter/opscenter    18563 2012-04-17 11:48 opscenter-agent/LICENSE

as you suggested, if you remove opscenter-agent-2.6-standalone.jar, the agent starts up fine.

Posted 1 year ago #

mbulman
Key Master

Thanks for the feedback. I assume you were upgrading from a previous installation? It sounds like there is a bug with not cleaning up if so. We'll look into it.

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.