We have our cassandra implementations setup so that nodes have two network interfaces, one for the gossip/jmx and one for thrift/app connections. How can I configure OpsCenter to use different IP addresses for thrift connections and jmx connections? As far as I can tell, you only specify a single IP address for cassandra nodes
In an effort to consolidate free help offered for our products we have decided to move these forums to a more widely used forum. Please use one of the following queries (or any combination):
- Cassandra: tag search or plain text search
- DataStax Enterprise: tag search or plain text search
- DataStax OpsCenter: tag search or plain text search
OpsCenter configuration for cassandra using dual networks(5 posts) (3 voices)
The agents use localhost for JMX by default, so you shouldn't have to change anything to set up opscenter like that. If using localhost is a problem, you can set "jmx_host" in address.yaml on each node to the IP that you need.
It appears that the agents are connecting to the localhost jmx/thrift ok, but opscenter is trying to communicate to the agents on the gossip interface of our cassandra cluster instead of the thrift/rpc interface. This is causing half of the information in opscenter to be missing (storage, etc) even though all the agents are showing connected.
here is the opscenterd log output. It SHOULD be connecting to 172.23.58.80, etc, the gossip network (172.23.1.x) is unroutable on purpose, so the opscenter server can not access it.
From what it seems, you are gathering up the node IPs via thrift calls, but that call is using autoDiscoverHosts instead of requesting the thrift IP address of the nodes, so it is defaulting to the gossip interface.
What I setup in opscenter for the seed addresses:
Output from opscenterd trying to connect to an agent:
WARN: HTTP request https://172.23.1.87:61621/cluster/datacenter?node_ip=172.23.1.81 failed: Connection was refused by other side: 111: Connection refused.
Some information on the 3 interfaces/IPs involved on the agent side can be seen in step 3 here: http://www.datastax.com/docs/opscenter/configure/configure_multi_region
It sounds like, since the gossip network is unroutable, you'll want to configure agent_rpc_interface on each node to point to cassandra's rpc interface (by default, that will pull from the broadcast address configured in cassandra.yaml)
If you're not able to bind to that interface on the agent side, you can configure agent_rpc_broadcast_address instead, which will allow the agent's http server to bind on a separate interface from the one opscenterd connects on.
Hope that helps. Let us know if you have any other questions.
This did help, I was able to successfully configure the agents for bi-directional communication. I do see another issue that I will open a new topic for. Thanks again!