cooptron Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 19:18:14 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-8000 Mon, 17 Dec 2012 21:26:13 +0000 cooptron 8000@http://www.datastax.com/support-forums/ <p>We do have secondary indexes, I assume that would affect the repair details as well since it is depending on a compaction? We will look forward to the new version! </p> <p>Thanks,<br /> Andrew </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7999 Mon, 17 Dec 2012 19:44:37 +0000 nickmbailey 7999@http://www.datastax.com/support-forums/ <p>Andrew,</p> <p>Do any of your CFs have secondary indexes? There is a known bug where secondary index compaction tasks can cause the compaction/streaming details to stall indefinitely. That will be fixed in the upcoming 2.1.3 release.</p> <p>-Nick </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7998 Mon, 17 Dec 2012 19:33:03 +0000 cooptron 7998@http://www.datastax.com/support-forums/ <p>The symptoms are the repair/compaction/stream information on the cluster views gets "stuck". The percentages no longer move, existing repairs do not go away in Opscenter, even though the cassandra node is no longer repairing or compacting. No new information shows up in opscenter for nodes that start repairs. Basically that part of the agent appears to stall indefinitely, while the OS stats and basic ring information still works.</p> <p>The only ERROR line I show in the agent log is from the initial configuration, which happens on every restart and appears to be the auto-discover process of the thrift port. It does connect to jmx on localhost</p> <p>ERROR [Initialization] 2012-12-17 12:24:01,871 MARK HOST AS DOWN TRIGGERED for host 10.1.1.43(10.1.1.43):9160<br /> ERROR [Initialization] 2012-12-17 12:24:01,872 Pool state on shutdown: &lt;ConcurrentCassandraClientPoolByHost&gt;:{10.1.1.43(10.1.1.43):9160}; IsActive?: true; Active: 0; Blocked: 0; Idle: 0; NumBeforeExhausted: 1<br /> ERROR [Initialization] 2012-12-17 12:24:01,878 Error when performing thrift operation: #&lt;HectorException me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.&gt;<br /> ERROR [Thread-5] 2012-12-17 12:24:01,879 Unable to connect to Cassandra #&lt;HectorException me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.&gt;</p> <p>I turned logging up to debugging on our test cluster (we are seeing the same situation there, much less CF's) and I see it collecting metrics on a regular basis, but then randomly it will spam the following (multiple times a second). Debugging level logging did not provide any additional information to the cause. I can send full logs if you are interested.</p> <p> WARN [Thread-2] 2012-12-17 13:16:13,062 Thrift operation queue is full, discarding thrift operation<br /> WARN [Thread-2] 2012-12-17 13:16:13,062 271315 operations dropped so far.</p> <p>It doesnt appear that the issue is based on number of metrics, the "stall" happens in our test cluster with roughly 100 CF's, and I cranked down the metrics in production (using ignore_keyspaces in the opscenter server config), but the issue still exists. </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7996 Mon, 17 Dec 2012 17:50:58 +0000 nickmbailey 7996@http://www.datastax.com/support-forums/ <p>Andrew,</p> <p>You are correct that reducing the column families you collect metrics for will help with thrift operations being discarded. Node information like compaction and streams is already separate from metric collection however. Those operations being discarded shouldn't be affecting that data. You are seeing compactions/streaming from nodetool that aren't showing up in OpsCenter?</p> <p>Are there any other errors in the agent log when you see this issue?</p> <p>-Nick </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7995 Mon, 17 Dec 2012 16:58:14 +0000 cooptron 7995@http://www.datastax.com/support-forums/ <p>We are seeing an issue with the opscenter agent where we stop receiving information related to cassandra activities in the opscenter server view (repairs, streams, compactions). We do still see IO information and load information. This correlates to log info on the agents stating that the thrift operations queue is full and it is dropping thrift requests. If we restart the agent it will start sending all information again for a limited timeframe and then starts dropping thrift operations again. We have quite a few column families (across all keyspaces, roughly 3000). I know we could likely fix this by reducing the amount of column families that we want to see metrics for, but I was curious if there were some tuning knobs to either increase the polling interval between metrics gatherings, or increase the thrift queue? Is there a way to put the operations information (repairs, compactions, streams) into a separate queue so it is not affected by the metrics gathering?</p> <p>Thanks,<br /> Andrew </p> http://www.datastax.com/support-forums/topic/opscenter-configuration-for-cassandra-using-dual-networks#post-7994 Mon, 17 Dec 2012 16:51:54 +0000 cooptron 7994@http://www.datastax.com/support-forums/ <p>This did help, I was able to successfully configure the agents for bi-directional communication. I do see another issue that I will open a new topic for. Thanks again! </p> http://www.datastax.com/support-forums/topic/opscenter-configuration-for-cassandra-using-dual-networks#post-7888 Fri, 07 Dec 2012 14:54:23 +0000 mbulman 7888@http://www.datastax.com/support-forums/ <p>Some information on the 3 interfaces/IPs involved on the agent side can be seen in step 3 here: <a href="http://www.datastax.com/docs/opscenter/configure/configure_multi_region" rel="nofollow">http://www.datastax.com/docs/opscenter/configure/configure_multi_region</a></p> <p>It sounds like, since the gossip network is unroutable, you'll want to configure agent_rpc_interface on each node to point to cassandra's rpc interface (by default, that will pull from the broadcast address configured in cassandra.yaml)</p> <p>If you're not able to bind to that interface on the agent side, you can configure agent_rpc_broadcast_address instead, which will allow the agent's http server to bind on a separate interface from the one opscenterd connects on.</p> <p>Hope that helps. Let us know if you have any other questions. </p> http://www.datastax.com/support-forums/topic/opscenter-configuration-for-cassandra-using-dual-networks#post-7868 Thu, 06 Dec 2012 23:07:42 +0000 cooptron 7868@http://www.datastax.com/support-forums/ <p>tupton,</p> <p>It appears that the agents are connecting to the localhost jmx/thrift ok, but opscenter is trying to communicate to the agents on the gossip interface of our cassandra cluster instead of the thrift/rpc interface. This is causing half of the information in opscenter to be missing (storage, etc) even though all the agents are showing connected.</p> <p>here is the opscenterd log output. It SHOULD be connecting to 172.23.58.80, etc, the gossip network (172.23.1.x) is unroutable on purpose, so the opscenter server can not access it. </p> <p>From what it seems, you are gathering up the node IPs via thrift calls, but that call is using autoDiscoverHosts instead of requesting the thrift IP address of the nodes, so it is defaulting to the gossip interface.</p> <p>What I setup in opscenter for the seed addresses:<br /> 172.23.58.80<br /> 172.23.58.81<br /> 172.23.58.82</p> <p>Output from opscenterd trying to connect to an agent:</p> <p>WARN: HTTP request <a href="https://172.23.1.87:61621/cluster/datacenter?node_ip=172.23.1.81" rel="nofollow">https://172.23.1.87:61621/cluster/datacenter?node_ip=172.23.1.81</a> failed: Connection was refused by other side: 111: Connection refused. </p> http://www.datastax.com/support-forums/topic/opscenter-configuration-for-cassandra-using-dual-networks#post-7014 Thu, 18 Oct 2012 20:18:00 +0000 tupton 7014@http://www.datastax.com/support-forums/ <p>cooptron,</p> <p>The agents use localhost for JMX by default, so you shouldn't have to change anything to set up opscenter like that. If using localhost is a problem, you can set "jmx_host" in address.yaml on each node to the IP that you need. </p> http://www.datastax.com/support-forums/topic/opscenter-configuration-for-cassandra-using-dual-networks#post-7013 Thu, 18 Oct 2012 20:08:46 +0000 cooptron 7013@http://www.datastax.com/support-forums/ <p>We have our cassandra implementations setup so that nodes have two network interfaces, one for the gossip/jmx and one for thrift/app connections. How can I configure OpsCenter to use different IP addresses for thrift connections and jmx connections? As far as I can tell, you only specify a single IP address for cassandra nodes </p>