DataStax Support Forums » User Favorites: mze

DataStax Support Forums » User Favorites: mze mze Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 06:48:57 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php mbulman on "Opscenter in NAT environment" http://www.datastax.com/support-forums/topic/opscenter-in-nat-environment#post-8974 Thu, 07 Feb 2013 20:03:49 +0000 mbulman 8974@http://www.datastax.com/support-forums/ Glad you got this figured out. We'll make sure the documentation is clearer on what to do for that option. Thanks for the feedback. mze on "Opscenter in NAT environment" http://www.datastax.com/support-forums/topic/opscenter-in-nat-environment#post-8939 Wed, 06 Feb 2013 09:19:34 +0000 mze 8939@http://www.datastax.com/support-forums/ Ah figured it out! agent_rpc_interface: 10.0.2.111 (local ip address, this is used by jetty as the bind address for the agent rpc webserver) local_interface: [public ip] (this is the ip that's used by opscenter to recognize the node, if you set this to the local ip opscenter doesn't recognize the node) So beside the stomp_interface setting these where the only 2 options that are in my agent config files. I did not use agent_rpc_broadcast_address. So it seems that the documentation on local_interface needs to be extended. The local_interface option is also used by opscenterd to recognize the node. Is this correct? Because when I set it to the local ip (10.0.2.111) I get: "WARN: Unable to find a matching cluster for [u'10.0.2.111']" in opscenterd.log. Despite setting agent_rpc_broadcast_address to the public ip. This probably does mean that an agent uses the nodes public ip to connect to it (while it is actually on the same machine). But I guess with the low amount of traffic through the agents this probably doesn't matter. tupton on "Opscenter in NAT environment" http://www.datastax.com/support-forums/topic/opscenter-in-nat-environment#post-8922 Tue, 05 Feb 2013 17:04:04 +0000 tupton 8922@http://www.datastax.com/support-forums/ mze, Have you tried keeping <code>local_interface</code> set to the local ip and setting <code>agent_rpc_broadcast_address</code> as appropriate? See <a href="http://www.datastax.com/docs/opscenter/configure/configure_multi_region" rel="nofollow">http://www.datastax.com/docs/opscenter/configure/configure_multi_region</a> . If you have seen that documentation before and are still having issues, or if I am not fully understanding your issue, please respond back so I can try to help you further. mze on "Opscenter in NAT environment" http://www.datastax.com/support-forums/topic/opscenter-in-nat-environment#post-8920 Tue, 05 Feb 2013 12:13:08 +0000 mze 8920@http://www.datastax.com/support-forums/ Hi there, I have some problems setting up opscenter in a cassandra cluster where the cassandra cluster is running in a NAT environment. The situation: The cassandra cluster consists of 3 nodes in a private network (10.0.0.0/16). Opscenter is also running in a private network which cannot be reached internally from the cassandra nodes. The cassandra nodes can communicate correctly to each other: nodetool gossipinfo /123.123.123.121 (public) INTERNAL_IP:10.0.0.211 STATUS:NORMAL,56713727820156407428984779325531226112 HOST_ID:d9cede23-5dd6-4d54-a17b-33fdc4d36f1f DC:eu-west SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8 RPC_ADDRESS:10.0.0.211 LOAD:98049.0 NET_VERSION:6 RELEASE_VERSION:1.2.1 SEVERITY:0.0 RACK:1b /123.123.123.122 (public) INTERNAL_IP:10.0.1.250 HOST_ID:da5ab1bd-8976-41eb-ab9c-a7b6b6a69a21 DC:eu-west SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8 RPC_ADDRESS:10.0.1.250 LOAD:98012.0 NET_VERSION:6 RELEASE_VERSION:1.2.1 SEVERITY:0.0 RACK:1a /123.123.123.123 (public) INTERNAL_IP:10.0.2.111 STATUS:NORMAL,113427455640312814857969558651062452224 HOST_ID:ef43d159-3220-4903-9eb4-2eacc04923b1 DC:eu-west SCHEMA:025bdec8-57c1-3268-9ef0-cc84ac90adb8 RPC_ADDRESS:10.0.2.111 LOAD:97977.0 NET_VERSION:6 RELEASE_VERSION:1.2.1 SEVERITY:0.0 RACK:1 The relevant settings in cassandra.yaml: listen_address: <internal ip> (10.0.2.111 for example) broadcast_address: <public ip> (123.123.123.123 for example) rpc_address: <internal ip> (10.0.2.111 for example) seeds: "10.0.1.250,10.0.0.211,10.0.2.111" Now the settings in address.yaml (agent config): stomp_interface: 123.123.123.124 (public ip of opscenter) rpc_interface: "10.0.2.111" (local ip) agent_rpc_interface: "123.123.123.123" (public ip of node) local_interface: "10.0.2.111" (local ip) use_ssl: 0 When I run the agents with this config on the opscenterd.log I get the following messages: WARN: Unable to find a matching cluster for [u'10.0.2.111'] When I change local_interface to the public ip so: stomp_interface: 123.123.123.124 (public ip of opscenter) rpc_interface: "10.0.2.111" (local ip) agent_rpc_interface: "123.123.123.123" (public ip of node) local_interface: "123.123.123.123" (public ip of node) use_ssl: 0 The opscenterd.log says: 2013-02-05 11:22:17+0000 [CommonSense_Development_cluster] INFO: Agent for ip 123.123.123.123 is version u'2.11' 2013-02-05 11:22:18+0000 [CommonSense_Development_cluster] INFO: Using 123.123.123.123 as the RPC address for node 123.123.123.123 So all seems well but if I now look in the opscenter agent log (of node 123.123.123.123) the following problems occurred: ERROR [Jetty] 2013-02-05 11:24:28,971 Exception running Jetty, restarting: java.net.BindException: Cannot assign requested address So it seems that Jetty uses the local_interface address as its bind ip address. Because the node is in a NAT environment it cannot bind to the public ip. I think the local_interface option should be separated from the jetty bind address. For the jetty bind address it would probably be better to use the rpc_interface option instead. Or add an override for it so that the jetty bind address can be set separately. Or maybe I'm overlooking something. Preferably I would like to keep the communication between the cassandra nodes on local network. Communication is a lot cheaper and faster on the local network. When I try autoconfig the only configuration options set in the agent config is the stomp_interface option: stomp_interface: 10.226.222.146 (private ip of opscenter)