Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

JDBC pool eating connections when using Thrift server

(7 posts) (2 voices)

Started 1 year ago by Marius Waldal
Latest reply from Marius Waldal
This topic is not resolved

Tags:

Marius Waldal
Member

Ok guys, we are at a loss here and hoping you may shed some light on this total show stopper!

We are using CQL v. 1.0.5 and are experiencing problems using jdbc pooling from a thrift server.

In our Core module, we have import jobs and calculation jobs using the thread pool without problems, reading and writing tens of thousands of records lightning fast.

However; we are also using a Server module (based on org.apache.thrift.server.TServer) which is in turn using the same code from Core for connections to Cassandra. When we use this server, it seems connections are not released and reused, but the pool is constantly growing until reaching a roof at <16.000 connections, after which we get a
Error in thread "Thread-3"
message on the server, and a timeout on the client.

What beats us is why this works fine when running the import and calculation jobs, but not when the same dao classes are used from the server (imported as a maven artifact).

Any ideas whatsoever would be greatly appreciated!

Cheers
Marius

Posted 1 year ago #
zznate

What pooling library are you using?

Do you have any examples of how you are using TServer to construct and manage connections? It sounds like you are just creating new connections instead of returning existing ones. I can't really speculate further than that without more information.

Posted 1 year ago #
Marius Waldal
Member

@zznate Thanks for answering.

The TServer is not managing the connections itself. It uses our Core module, which has a Spring-configured datasource that extends org.apache.tomcat.jdbc.pool.DataSource.
maxActive is set to 50, and we see during startup that this is read by the server. However, it seems that the connections are never returned when used by the server module, although this is not a problem when Core itself uses it for import and calculation.

The TServer is using an org.apache.commons.pool.impl.GenericObjectPool for its workers, but as far as we can tell, this is not the pool that eats threads.

Posted 1 year ago #
zznate

How are you constructing the underlying TSocket implementation and with which Transport (if any) are you wrapping said TSocket?

ie. the call to DataSource#getConnection() should hit a pool who's contents have been constructed thusly:

ttps://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/client/HThriftClient.java#L123-146

Side note: after extensive testing in hector over a year ago, we *ripped out* commons pooling due to poor concurrency characteristics with lots of threads. YMMV though, as we had specific general-purpose targets for which we were aiming.

Oh, it's not quite ready for prime-time, but we will be offering https://github.com/riptano/jdbc-conn-pool shortly.

Posted 1 year ago #
Marius Waldal
Member

The TSocket is wrapped by TTransport. The socket is created like this:

TTransport createTSocket(final String host, final int port) { final TSocket tsocket = new TSocket(host, port, socketTimeout); try { tsocket.getSocket().setKeepAlive(false); } catch (SocketException ex) { logger.warn("failed to disable keepalive"); } return tsocket; }

This is done inside TSocketFactory; a class creating TSockets based on a UriManager for managing multiple hosts, and the makeObject() method here is used by GenericObjectPool.borrowObject()

Was this answering your question?

Thanks for the sidenote. I will take this up with our other developers. And looking forward to the jdbc-conn-pool !

Posted 1 year ago #
zznate

If I am reading that correctly, every time you invoke GenericObjectPool#borrowObject you are creating a new socket instead of returning an existing socket from the pool. Is GOP#releaseObject invoked with the socket when the connection usage is finished?

This really just sounds like a misunderstanding with the commons-pooling API. I would suggest a look at DBCP in commons (which uses commons pooling for vanilla JDBC usage for some extra examples with a similar use case.

Posted 1 year ago #
Marius Waldal
Member

We have switched communication from a local cassandra to a dev server now, and here the problem is only when using JDBC connections against cassandra. If we switch to Hector, the problem does not persist.

We have decided, for now, to stick with Hector for the connections from the Thrift server and use JDBC for Core operations. We will return to the JDBC issues after the system is live using Hector and everything works well.

Posted 1 year ago #

RSS feed for this topic