http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 03:01:15 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-1921 Wed, 16 May 2012 01:53:40 +0000 Anonymous 1921@http://www.datastax.com/support-forums/ <p><a href="http://www.clearance-polo-shirts.com" title="polo shirts outlet online">polo shirts outlet online</a> are labeled "polos" as well as became therefore popular on the game of golf classes; persons started phoning all of them the game of golf tops. Even though terms "<a href="http://www.clearance-polo-shirts.com" title="abercrombie polo outlet online">abercrombie polo outlet online</a> " as well as "golf shirt" are employed interchangeably, the idea "polo shirt" will be popular. Embroidered <a href="http://www.clearance-polo-shirts.com" title="lacoste polo shirts outlet store">lacoste polo shirts outlet store</a> are put on intended for company, everyday as well as leisure activities<a href="http://www.clearance-polo-shirts.com">http://www.clearance-polo-shirts.com</a>. </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-414 Tue, 09 Aug 2011 18:00:20 +0000 zznate 414@http://www.datastax.com/support-forums/ <p>rkreznarich,<br /> See the post above - is it possible you are sending a payload larger than 16mb in a single operation?</p> <p>The general issue here is that a socket error usually only results when a Cassandra server hits the wall. We probably take an over-agressive approach here on removing the pool after receiving the first error. I'll take a look and see if there is something quick here that we can get out. </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-413 Tue, 09 Aug 2011 17:58:03 +0000 zznate 413@http://www.datastax.com/support-forums/ <p>Anthony,<br /> You should not have to delete any data in order to change this parameter. For reference though, see the following guide on backups:<br /> <a href="http://www.datastax.com/docs/0.8/operations/scheduled_tasks#backing-up-data" rel="nofollow">http://www.datastax.com/docs/0.8/operations/scheduled_tasks#backing-up-data</a></p> <p>Thanks for the additional info though. Let me know how it goes. </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-412 Mon, 08 Aug 2011 17:42:33 +0000 rkreznarich 412@http://www.datastax.com/support-forums/ <p>Hi Nate, We put in the change to go to Hector 0.8.0-2 and set the useSocketKeepalive = true in our startup configuration, but it had no effect. </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-411 Mon, 08 Aug 2011 17:25:19 +0000 anthony.ikeda 411@http://www.datastax.com/support-forums/ <p>Nate, I raised this on the Cassandra User mailing list about a week ago with what I just discovered the following resolution.</p> <p>It seems Tim Snyder recommended increasing the size value of:</p> <p># The max length of a thrift message, including all fields and<br /> # internal thrift overhead.<br /> thrift_max_message_length_in_mb: 16</p> <p>Which seems to have fixed the problem for him. He also had to delete the data stores in order for it to work.</p> <p>Are there any scripts to back up the data we have so it can be restored once we have changed this setting? Or do we need to manually use the sstable2json to perform this task on each column family data file?</p> <p>Anthony </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-408 Fri, 05 Aug 2011 21:38:52 +0000 rkreznarich 408@http://www.datastax.com/support-forums/ <p>Hector -&gt; 0.8.0-1<br /> Cassandra -&gt; 0.8.0</p> <p>I will add the useSocketKeepalive in our configuration and try that out.</p> <p>Thanks for your reply. </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-406 Fri, 05 Aug 2011 01:24:00 +0000 zznate 406@http://www.datastax.com/support-forums/ <p>What version of Hector I you using? If you are not already, please upgrade to either 0.7.0-31 or 0.8.0-2 depending on which version of Cassandra you are using (0.7.x or 0.8.x respectively).</p> <p>In both versions it might be advisable to enable keep-alive via CassandraHostConfigurator#setUseSocketKeepalive </p> http://www.datastax.com/support-forums/topic/losing-connections-to-cassandra#post-405 Thu, 04 Aug 2011 22:48:49 +0000 rkreznarich 405@http://www.datastax.com/support-forums/ <p>We are experiencing an issue where we are losing Cassandra connections intermittently. Here is what happens in the logs:</p> <p>(specific host footprint information removed from log file)</p> <p>A request comes in and we get this ERROR entry ...</p> <p>ERROR [me.prettyprint.cassandra.connection.HThriftClient] - Could not flush transport (to be expected if the pool is shutting down) in close for client: XXXX<br /> org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe</p> <p>At the end of this exception chain, the pool is marked down ...</p> <p>ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - MARK HOST AS DOWN TRIGGERED for host XXXX<br /> ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - Pool state on shutdown: &lt;ConcurrentCassandraClientPoolByHost&gt;:{XXXX}; IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted: 49<br /> ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown triggered on &lt;ConcurrentCassandraClientPoolByHost&gt;:{XXXX}<br /> ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown complete on &lt;ConcurrentCassandraClientPoolByHost&gt;:{XXXX}<br /> INFO [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Host detected as down was added to retry queue: XXXX<br /> WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Could not fullfill request on this host CassandraClient&lt;XXXX&gt;</p> <p>... and it appears as if the connection has been lost.</p> <p>Eventually a new incoming request will reset the connection and the connection pool will be reestablished, but that initial incoming request fails.<br /> Subsequent requests work ok. </p> <p>The connections are reestablished ...</p> <p>INFO [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Downed Host retry status true with host: XXXX<br /> INFO [me.prettyprint.cassandra.connection.HConnectionManager] - Added host XXXX to pool<br /> INFO [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Downed Host retry status true with host: XXXX<br /> INFO [me.prettyprint.cassandra.connection.HConnectionManager] - Added host XXXX to pool</p> <p>Everything works ok for awhile again. </p>