http://www.datastax.com/support-forums/tags/thrift Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 22:49:58 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/cql-question#post-1663 Tue, 17 Apr 2012 21:16:04 +0000 thepaul 1663@http://www.datastax.com/support-forums/ <p>The version of the Thrift library in use (Thrift.php and friends) is more important, and is not necessarily the same as the version of Thrift you used to compile your interface.</p> <p>Are you using cassandra-pdo for your CQL queries, or some other library? What version? </p> http://www.datastax.com/support-forums/topic/cql-question#post-1646 Tue, 17 Apr 2012 08:03:57 +0000 Anonymous 1646@http://www.datastax.com/support-forums/ <p>We running with an interface compiled from Thrift 0.7.0 </p> http://www.datastax.com/support-forums/topic/cql-question#post-1629 Mon, 16 Apr 2012 16:41:41 +0000 thepaul 1629@http://www.datastax.com/support-forums/ <p>Hmm. I'm guessing that must be some sort of shortcoming in the PHP implementation of Thrift. I've just tested with a query string in excess of 2^20 bytes (1 MiB) through the Python thrift interface, and it worked fine.</p> <p>I asked around, and it sounds like the more recent versions of PHP Thrift should have a higher maximum frame size. What version are you using? </p> http://www.datastax.com/support-forums/topic/cql-question#post-1628 Mon, 16 Apr 2012 10:00:57 +0000 Anonymous 1628@http://www.datastax.com/support-forums/ <p>Hi there, we are just switching to CQL for some of our data analysis (we were using direct thrift queries) and I have a question which I'm guessing must have come up before. I have a large amount of user data in a super column so I have written a (PHP) script to read that data and push it into a wide column suitable for CQL.</p> <p>At the moment I have a loop that looks something like this:</p> <p>Read a chunk of keys from the super-column like so<br /> $cCQLResult = $this-&gt;ExecuteCQL($cCQLDatasource, "SELECT KEY FROM PlayrUserData WHERE KEY &gt;= '$StartKey' AND KEY &lt; '$FinishKey'");</p> <p>Now for that chunk build a list of keys that I wish to write (using a thrift query into the super column)</p> <p>For those keys that pass a simple test write them all into a wide column family:<br /> $this-&gt;ExecuteCQL($cCQLDatasource, "UPDATE MarketingCohortCache SET '000001.Facebook.1517061723'='1','000020.Facebook.100003474181964'='1', .. 'WHERE KEY=LapsedUser_1330560000");</p> <p>The problem appears to be that I am attempting to UPDATE MarketingCohortCache with a pretty long list (about 5000 keys) and it appears to be choking the thrift interface with: TSocket: Could not write 194718 bytes (the length of the Update string).</p> <p>Is there a recommended way to write a large quantity of data to a wide column? Do I need to loop through with multiple updates? If so how long can I let the update string get?</p> <p>May thanks for reading. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-914 Thu, 22 Dec 2011 13:28:59 +0000 Marius Waldal 914@http://www.datastax.com/support-forums/ <p>We have switched communication from a local cassandra to a dev server now, and here the problem is only when using JDBC connections against cassandra. If we switch to Hector, the problem does not persist.</p> <p>We have decided, for now, to stick with Hector for the connections from the Thrift server and use JDBC for Core operations. We will return to the JDBC issues after the system is live using Hector and everything works well. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-880 Tue, 20 Dec 2011 16:23:58 +0000 zznate 880@http://www.datastax.com/support-forums/ <p>If I am reading that correctly, every time you invoke GenericObjectPool#borrowObject you are creating a new socket instead of returning an existing socket from the pool. Is GOP#releaseObject invoked with the socket when the connection usage is finished?</p> <p>This really just sounds like a misunderstanding with the commons-pooling API. I would suggest a look at DBCP in commons (which uses commons pooling for vanilla JDBC usage for some extra examples with a similar use case. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-877 Tue, 20 Dec 2011 07:51:20 +0000 Marius Waldal 877@http://www.datastax.com/support-forums/ <p>The TSocket is wrapped by TTransport. The socket is created like this:</p> <p><code><br /> TTransport createTSocket(final String host, final int port) {<br /> final TSocket tsocket = new TSocket(host, port, socketTimeout);<br /> try {<br /> tsocket.getSocket().setKeepAlive(false);<br /> } catch (SocketException ex) {<br /> logger.warn("failed to disable keepalive");<br /> }<br /> return tsocket; }<br /> </code></p> <p>This is done inside TSocketFactory; a class creating TSockets based on a UriManager for managing multiple hosts, and the makeObject() method here is used by GenericObjectPool.borrowObject()</p> <p>Was this answering your question?</p> <p>Thanks for the sidenote. I will take this up with our other developers. And looking forward to the jdbc-conn-pool ! </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-872 Mon, 19 Dec 2011 21:20:51 +0000 zznate 872@http://www.datastax.com/support-forums/ <p>How are you constructing the underlying TSocket implementation and with which Transport (if any) are you wrapping said TSocket?</p> <p>ie. the call to DataSource#getConnection() should hit a pool who's contents have been constructed thusly:</p> <p> ttps://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/client/HThriftClient.java#L123-146</p> <p>Side note: after extensive testing in hector over a year ago, we *ripped out* commons pooling due to poor concurrency characteristics with lots of threads. YMMV though, as we had specific general-purpose targets for which we were aiming.</p> <p>Oh, it's not quite ready for prime-time, but we will be offering <a href="https://github.com/riptano/jdbc-conn-pool" rel="nofollow">https://github.com/riptano/jdbc-conn-pool</a> shortly. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-871 Mon, 19 Dec 2011 21:05:33 +0000 Marius Waldal 871@http://www.datastax.com/support-forums/ <p>@zznate Thanks for answering.</p> <p>The TServer is not managing the connections itself. It uses our Core module, which has a Spring-configured datasource that extends org.apache.tomcat.jdbc.pool.DataSource.<br /> maxActive is set to 50, and we see during startup that this is read by the server. However, it seems that the connections are never returned when used by the server module, although this is not a problem when Core itself uses it for import and calculation.</p> <p>The TServer is using an org.apache.commons.pool.impl.GenericObjectPool for its workers, but as far as we can tell, this is not the pool that eats threads. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-868 Mon, 19 Dec 2011 17:20:21 +0000 zznate 868@http://www.datastax.com/support-forums/ <p>What pooling library are you using?</p> <p>Do you have any examples of how you are using TServer to construct and manage connections? It sounds like you are just creating new connections instead of returning existing ones. I can't really speculate further than that without more information. </p> http://www.datastax.com/support-forums/topic/jdbc-pool-eating-connections-when-using-thrift-server#post-867 Mon, 19 Dec 2011 13:34:11 +0000 Marius Waldal 867@http://www.datastax.com/support-forums/ <p>Ok guys, we are at a loss here and hoping you may shed some light on this total show stopper!</p> <p>We are using CQL v. 1.0.5 and are experiencing problems using jdbc pooling from a thrift server.</p> <p>In our Core module, we have import jobs and calculation jobs using the thread pool without problems, reading and writing tens of thousands of records lightning fast.</p> <p>However; we are also using a Server module (based on org.apache.thrift.server.TServer) which is in turn using the same code from Core for connections to Cassandra. When we use this server, it seems connections are not released and reused, but the pool is constantly growing until reaching a roof at &lt;16.000 connections, after which we get a<br /> Error in thread "Thread-3"<br /> message on the server, and a timeout on the client.</p> <p>What beats us is why this works fine when running the import and calculation jobs, but not when the same dao classes are used from the server (imported as a maven artifact). </p> <p>Any ideas whatsoever would be greatly appreciated!</p> <p>Cheers<br /> Marius </p>