Hi there,
I have some problems setting up opscenter in a cassandra cluster where the cassandra cluster is running in a NAT environment.
The situation:
The cassandra cluster consists of 3 nodes in a private network (10.0.0.0/16). Opscenter is also running in a private network which cannot be reached internally from the cassandra nodes. The cassandra nodes can communicate correctly to each other:
nodetool gossipinfo
/123.123.123.121 (public)
INTERNAL_IP:10.0.0.211
STATUS:NORMAL,56713727820156407428984779325531226112
HOST_ID:d9cede23-5dd6-4d54-a17b-33fdc4d36f1f
DC:eu-west
SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8
RPC_ADDRESS:10.0.0.211
LOAD:98049.0
NET_VERSION:6
RELEASE_VERSION:1.2.1
SEVERITY:0.0
RACK:1b
/123.123.123.122 (public)
INTERNAL_IP:10.0.1.250
HOST_ID:da5ab1bd-8976-41eb-ab9c-a7b6b6a69a21
DC:eu-west
SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8
RPC_ADDRESS:10.0.1.250
LOAD:98012.0
NET_VERSION:6
RELEASE_VERSION:1.2.1
SEVERITY:0.0
RACK:1a
/123.123.123.123 (public)
INTERNAL_IP:10.0.2.111
STATUS:NORMAL,113427455640312814857969558651062452224
HOST_ID:ef43d159-3220-4903-9eb4-2eacc04923b1
DC:eu-west
SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8
RPC_ADDRESS:10.0.2.111
LOAD:97977.0
NET_VERSION:6
RELEASE_VERSION:1.2.1
SEVERITY:0.0
RACK:1
The relevant settings in cassandra.yaml:
listen_address: <internal ip> (10.0.2.111 for example)
broadcast_address: <public ip> (123.123.123.123 for example)
rpc_address: <internal ip> (10.0.2.111 for example)
seeds: "10.0.1.250,10.0.0.211,10.0.2.111"
Now the settings in address.yaml (agent config):
stomp_interface: 123.123.123.124 (public ip of opscenter)
rpc_interface: "10.0.2.111" (local ip)
agent_rpc_interface: "123.123.123.123" (public ip of node)
local_interface: "10.0.2.111" (local ip)
use_ssl: 0
When I run the agents with this config on the opscenterd.log I get the following messages:
WARN: Unable to find a matching cluster for [u'10.0.2.111']
When I change local_interface to the public ip so:
stomp_interface: 123.123.123.124 (public ip of opscenter)
rpc_interface: "10.0.2.111" (local ip)
agent_rpc_interface: "123.123.123.123" (public ip of node)
local_interface: "123.123.123.123" (public ip of node)
use_ssl: 0
The opscenterd.log says:
2013-02-05 11:22:17+0000 [CommonSense_Development_cluster] INFO: Agent for ip 123.123.123.123 is version u'2.11'
2013-02-05 11:22:18+0000 [CommonSense_Development_cluster] INFO: Using 123.123.123.123 as the RPC address for node 123.123.123.123
So all seems well but if I now look in the opscenter agent log (of node 123.123.123.123) the following problems occurred:
ERROR [Jetty] 2013-02-05 11:24:28,971 Exception running Jetty, restarting: java.net.BindException: Cannot assign requested address
So it seems that Jetty uses the local_interface address as its bind ip address.
Because the node is in a NAT environment it cannot bind to the public ip.
I think the local_interface option should be separated from the jetty bind address. For the jetty bind address it would probably be better to use the rpc_interface option instead. Or add an override for it so that the jetty bind address can be set separately.
Or maybe I'm overlooking something. Preferably I would like to keep the communication between the cassandra nodes on local network. Communication is a lot cheaper and faster on the local network.
When I try autoconfig the only configuration options set in the agent config is the stomp_interface option:
stomp_interface: 10.226.222.146 (private ip of opscenter)
