http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 03:11:41 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-8000 Mon, 17 Dec 2012 21:26:13 +0000 cooptron 8000@http://www.datastax.com/support-forums/ <p>We do have secondary indexes, I assume that would affect the repair details as well since it is depending on a compaction? We will look forward to the new version! </p> <p>Thanks,<br /> Andrew </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7999 Mon, 17 Dec 2012 19:44:37 +0000 nickmbailey 7999@http://www.datastax.com/support-forums/ <p>Andrew,</p> <p>Do any of your CFs have secondary indexes? There is a known bug where secondary index compaction tasks can cause the compaction/streaming details to stall indefinitely. That will be fixed in the upcoming 2.1.3 release.</p> <p>-Nick </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7998 Mon, 17 Dec 2012 19:33:03 +0000 cooptron 7998@http://www.datastax.com/support-forums/ <p>The symptoms are the repair/compaction/stream information on the cluster views gets "stuck". The percentages no longer move, existing repairs do not go away in Opscenter, even though the cassandra node is no longer repairing or compacting. No new information shows up in opscenter for nodes that start repairs. Basically that part of the agent appears to stall indefinitely, while the OS stats and basic ring information still works.</p> <p>The only ERROR line I show in the agent log is from the initial configuration, which happens on every restart and appears to be the auto-discover process of the thrift port. It does connect to jmx on localhost</p> <p>ERROR [Initialization] 2012-12-17 12:24:01,871 MARK HOST AS DOWN TRIGGERED for host 10.1.1.43(10.1.1.43):9160<br /> ERROR [Initialization] 2012-12-17 12:24:01,872 Pool state on shutdown: &lt;ConcurrentCassandraClientPoolByHost&gt;:{10.1.1.43(10.1.1.43):9160}; IsActive?: true; Active: 0; Blocked: 0; Idle: 0; NumBeforeExhausted: 1<br /> ERROR [Initialization] 2012-12-17 12:24:01,878 Error when performing thrift operation: #&lt;HectorException me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.&gt;<br /> ERROR [Thread-5] 2012-12-17 12:24:01,879 Unable to connect to Cassandra #&lt;HectorException me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.&gt;</p> <p>I turned logging up to debugging on our test cluster (we are seeing the same situation there, much less CF's) and I see it collecting metrics on a regular basis, but then randomly it will spam the following (multiple times a second). Debugging level logging did not provide any additional information to the cause. I can send full logs if you are interested.</p> <p> WARN [Thread-2] 2012-12-17 13:16:13,062 Thrift operation queue is full, discarding thrift operation<br /> WARN [Thread-2] 2012-12-17 13:16:13,062 271315 operations dropped so far.</p> <p>It doesnt appear that the issue is based on number of metrics, the "stall" happens in our test cluster with roughly 100 CF's, and I cranked down the metrics in production (using ignore_keyspaces in the opscenter server config), but the issue still exists. </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7996 Mon, 17 Dec 2012 17:50:58 +0000 nickmbailey 7996@http://www.datastax.com/support-forums/ <p>Andrew,</p> <p>You are correct that reducing the column families you collect metrics for will help with thrift operations being discarded. Node information like compaction and streams is already separate from metric collection however. Those operations being discarded shouldn't be affecting that data. You are seeing compactions/streaming from nodetool that aren't showing up in OpsCenter?</p> <p>Are there any other errors in the agent log when you see this issue?</p> <p>-Nick </p> http://www.datastax.com/support-forums/topic/opscenter-agent-stuck-on-maxed-thrift-queue#post-7995 Mon, 17 Dec 2012 16:58:14 +0000 cooptron 7995@http://www.datastax.com/support-forums/ <p>We are seeing an issue with the opscenter agent where we stop receiving information related to cassandra activities in the opscenter server view (repairs, streams, compactions). We do still see IO information and load information. This correlates to log info on the agents stating that the thrift operations queue is full and it is dropping thrift requests. If we restart the agent it will start sending all information again for a limited timeframe and then starts dropping thrift operations again. We have quite a few column families (across all keyspaces, roughly 3000). I know we could likely fix this by reducing the amount of column families that we want to see metrics for, but I was curious if there were some tuning knobs to either increase the polling interval between metrics gatherings, or increase the thrift queue? Is there a way to put the operations information (repairs, compactions, streams) into a separate queue so it is not affected by the metrics gathering?</p> <p>Thanks,<br /> Andrew </p>