DataStax Support Forums » User Favorites: hliu

DataStax Support Forums » User Favorites: hliu hliu Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 11:52:18 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Brent on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10795 Wed, 08 May 2013 21:38:08 +0000 Brent 10795@http://www.datastax.com/support-forums/ At the bottom here: <a href="http://www.datastax.com/docs/datastax_enterprise3.0/reference/start_stop_dse" rel="nofollow">http://www.datastax.com/docs/datastax_enterprise3.0/reference/start_stop_dse</a> it says to run "nodetool drain" before "dse cassandra-stop". Seems like the "nodetool drain" within "dse cassandra-stop" should be enough, and I seem to recall even seeing DSE log messages saying something to the effect of "Already drained" when the second one comes in. Sven on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10784 Wed, 08 May 2013 19:31:32 +0000 Sven 10784@http://www.datastax.com/support-forums/ One drain should be enough, but it's possible that there is information out there that I don't have :-). Sven Brent on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10783 Wed, 08 May 2013 19:03:14 +0000 Brent 10783@http://www.datastax.com/support-forums/ Yeah, but should I drain it twice? this: nodetool drain nodetool disablethrift nodetool disablegossip nodetool drain kill -9 <DSE_PID> or this: nodetool disablethrift nodetool disablegossip nodetool drain kill -9 <DSE_PID> Sven on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10782 Wed, 08 May 2013 18:40:52 +0000 Sven 10782@http://www.datastax.com/support-forums/ In general it is a good idea to drain the node. It will allow for a faster start later. Of course that implies that the drain works properly. We are working on a fix for this issue. Thanks, Sven Brent on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10781 Wed, 08 May 2013 18:31:01 +0000 Brent 10781@http://www.datastax.com/support-forums/ I'm also having this problem, in both a single node and 3 node environment. Our procedure for stopping DSE is: nodetool drain -h <HOST_NAME> dse cassandra-stop -p <DSE_PID> It looks like within the dse script (DSE_HOME/bin/dse), these commands are running: nodetool disablethrift nodetool disablegossip nodetool drain kill -9 <DSE_PID> I've tried each nodetool command (disablethrift, disablegossip, drain) by themselves with a fresh DSE run, and disablethrift and drain cause DSE to block at the "Deactivating plugin: com.datastax.bdp.plugin.SolrContainerPlugin" log message. disablegossip completes successfully. When using DSE 3.0, the "nodetool drain" followed by "dse cassandra-stop" command sequence worked without issue, but since updating to 3.0.1, the "nodetool drain/disablethrift" commands block indefinitely. Related question -- is there any reason I'm running the "nodetool drain" command before I run "dse cassandra-stop"? I can't remember if I read to do that in some documentation or if one of my coworkers told me to do that, but it seems unnecessary since the drain command is run again in "dse cassandra-stop". sergio on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10576 Thu, 02 May 2013 22:46:23 +0000 sergio 10576@http://www.datastax.com/support-forums/ Thanks for the additional information. We've been able to reproduce and we're looking into it. Cheers, Sergio B. hliu on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10533 Tue, 30 Apr 2013 21:24:33 +0000 hliu 10533@http://www.datastax.com/support-forums/ ===== FULL THREAD DUMP =============== Tue Apr 30 14:20:07 2013 Oracle JRockit(R) R28.1.1-14-139783-1.6.0_22-20101206-0241-linux-x86_64 "(Signal Handler)" id=2 idx=0x8 tid=30025 prio=5 alive, native_blocked, daemon "(OC Main Thread)" id=3 idx=0xc tid=30026 prio=5 alive, native_waiting, daemon "(GC Worker Thread 1)" id=? idx=0x10 tid=30027 prio=5 alive, daemon "(GC Worker Thread 2)" id=? idx=0x14 tid=30028 prio=5 alive, daemon "(GC Worker Thread 3)" id=? idx=0x18 tid=30029 prio=5 alive, daemon "(GC Worker Thread 4)" id=? idx=0x1c tid=30030 prio=5 alive, daemon "(Code Generation Thread 1)" id=4 idx=0x20 tid=30031 prio=5 alive, native_waiting, daemon "(Code Optimization Thread 1)" id=5 idx=0x24 tid=30032 prio=5 alive, native_waiting, daemon "(VM Periodic Task)" id=6 idx=0x28 tid=30033 prio=10 alive, native_blocked, daemon "Finalizer" id=7 idx=0x2c tid=30034 prio=8 alive, native_waiting, daemon at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method) at jrockit/memory/Finalizer.access$700(Finalizer.java:12) at jrockit/memory/Finalizer$4.run(Finalizer.java:183) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "Reference Handler" id=8 idx=0x30 tid=30035 prio=10 alive, native_waiting, daemon at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method) at java/lang/ref/Reference.access$100(Reference.java:11) at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "(Sensor Event Thread)" id=9 idx=0x34 tid=30036 prio=5 alive, native_blocked, daemon "VM JFR Buffer Thread" id=10 idx=0x38 tid=30037 prio=5 alive, in native, daemon "JMAPI event thread" id=13 idx=0x3c tid=30038 prio=5 alive, in native, daemon "JFR request timer" id=14 idx=0x40 tid=30039 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: java/util/TaskQueue@0xc5e3f4b0[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Object.wait(Object.java:485) at java/util/TimerThread.mainLoop(Timer.java:483) ^-- Lock released while waiting: java/util/TaskQueue@0xc5e3f4b0[fat lock] at java/util/TimerThread.run(Timer.java:462) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "RMI TCP Accept-9290" id=16 idx=0x44 tid=30040 prio=5 alive, in native, daemon at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method) at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390) ^-- Holding lock: java/net/SocksSocketImpl@0xc5b94b00[biased lock] at java/net/ServerSocket.implAccept(ServerSocket.java:453) at java/net/ServerSocket.accept(ServerSocket.java:421) at sun/rmi/transport/tcp/TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun/rmi/transport/tcp/TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "RMI TCP Accept-0" id=17 idx=0x48 tid=30041 prio=5 alive, in native, daemon at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method) at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390) ^-- Holding lock: java/net/SocksSocketImpl@0xc5bf1858[biased lock] at java/net/ServerSocket.implAccept(ServerSocket.java:453) at java/net/ServerSocket.accept(ServerSocket.java:421) at oracle/jrockit/management/server/LocalJMXConnector$LocalRMIServerSocketFactory$1.accept(LocalJMXConnector.java:96) at sun/rmi/transport/tcp/TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun/rmi/transport/tcp/TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "(JDP autodiscovery)" id=18 idx=0x4c tid=30042 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: oracle/jrockit/management/server/jdp/JDPConnector@0xc61fb438[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at oracle/jrockit/management/server/jdp/JDPConnector.run(JDPConnector.java:222) ^-- Lock released while waiting: oracle/jrockit/management/server/jdp/JDPConnector@0xc61fb438[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "FileWatchdog" id=19 idx=0x50 tid=30043 prio=5 alive, sleeping, native_waiting, daemon at java/lang/Thread.sleep(J)V(Native Method) at org/apache/log4j/helpers/FileWatchdog.run(FileWatchdog.java:104) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "ScheduledTasks:1" id=20 idx=0x54 tid=30044 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "EXPIRING-MAP-REAPER:1" id=21 idx=0x58 tid=30045 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "OptionalTasks:1" id=22 idx=0x5c tid=30046 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "MemtablePostFlusher:1" id=24 idx=0x64 tid=30048 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-0" id=37 idx=0x98 tid=30061 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc232f338 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-1" id=38 idx=0x9c tid=30062 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc2332368 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-2" id=39 idx=0xa0 tid=30063 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc2332cb8 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-3" id=40 idx=0xa4 tid=30064 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc2333608 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-4" id=41 idx=0xa8 tid=30065 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc2333f58 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-5" id=42 idx=0xac tid=30066 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc23348a8 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-6" id=43 idx=0xb0 tid=30067 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc23351f8 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "IndexPool thread-7" id=44 idx=0xb4 tid=30068 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc2335b50 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at com/datastax/bdp/cassandra/index/solr/concurrent/IndexWorker.run(IndexWorker.java:32) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:441) at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303) at java/util/concurrent/FutureTask.run(FutureTask.java:138) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "pool-3-thread-1" id=45 idx=0xb8 tid=30069 prio=5 alive, parked, native_blocked at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "searcherExecutor-1-thread-1" id=47 idx=0xc0 tid=30071 prio=5 alive, parked, native_blocked -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc9b352c0 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "commitlog_archiver:1" id=48 idx=0xc4 tid=30073 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "COMMIT-LOG-ALLOCATOR" id=49 idx=0xc8 tid=30074 prio=5 alive, parked, native_blocked at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org/apache/cassandra/db/commitlog/CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:91) at org/apache/cassandra/utils/WrappedRunnable.run(WrappedRunnable.java:30) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "COMMIT-LOG-WRITER" id=50 idx=0xcc tid=30075 prio=5 alive, parked, native_blocked at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org/apache/cassandra/db/commitlog/PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:47) at org/apache/cassandra/utils/WrappedRunnable.run(WrappedRunnable.java:30) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "PERIODIC-COMMIT-LOG-SYNCER" id=51 idx=0xd0 tid=30076 prio=5 alive, sleeping, native_waiting at java/lang/Thread.sleep(J)V(Native Method) at org/apache/cassandra/db/commitlog/PeriodicCommitLogExecutorService$3.run(PeriodicCommitLogExecutorService.java:77) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "StreamStage:1" id=156 idx=0x274 tid=30181 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "GossipStage:1" id=157 idx=0x278 tid=30182 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "AntiEntropyStage:1" id=158 idx=0x27c tid=30183 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "MigrationStage:1" id=159 idx=0x280 tid=30184 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "MiscStage:1" id=160 idx=0x284 tid=30185 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "GossipTasks:1" id=169 idx=0x2a0 tid=30192 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "ACCEPT-localhost/127.0.0.1" id=170 idx=0x2a4 tid=30193 prio=5 alive, in native at sun/nio/ch/ServerSocketChannelImpl.accept0(Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;[Ljava/net/InetSocketAddress;)I(Native Method) at sun/nio/ch/ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) ^-- Holding lock: java/lang/Object@0xc7594dd0[biased lock] at sun/nio/ch/ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) ^-- Holding lock: java/lang/Object@0xc7594dc8[biased lock] at org/apache/cassandra/net/MessagingService$SocketThread.run(MessagingService.java:699) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "HintedHandoff:1" id=171 idx=0x2a8 tid=30194 prio=1 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "PendingRangeCalculator:1" id=173 idx=0x2b0 tid=30196 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "NonPeriodicTasks:1" id=174 idx=0x2b4 tid=30197 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc414e2a0 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/DelayQueue.take(DelayQueue.java:160) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "GC Daemon" id=176 idx=0x2bc tid=30205 prio=2 alive, waiting, native_blocked, daemon -- Waiting for notification on: sun/misc/GC$LatencyLock@0xc6999698[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at sun/misc/GC$Daemon.run(GC.java:100) ^-- Lock released while waiting: sun/misc/GC$LatencyLock@0xc6999698[fat lock] at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "ContainerBackgroundProcessor[StandardEngine[Solr]]" id=177 idx=0x2c0 tid=30206 prio=5 alive, sleeping, native_waiting, daemon at java/lang/Thread.sleep(J)V(Native Method) at org/apache/catalina/core/ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1627) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "TP-Processor1" id=178 idx=0x2c4 tid=30207 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48daa8[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Object.wait(Object.java:485) at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:662) ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48daa8[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "TP-Processor2" id=179 idx=0x2c8 tid=30208 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48def8[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Object.wait(Object.java:485) at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:662) ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48def8[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "TP-Processor3" id=180 idx=0x2cc tid=30209 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48e308[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Object.wait(Object.java:485) at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:662) ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0xce48e308[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "TP-Processor4" id=181 idx=0x2d0 tid=30210 prio=5 alive, in native, daemon at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method) at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390) ^-- Holding lock: java/net/SocksSocketImpl@0xce4448b0[biased lock] at java/net/ServerSocket.implAccept(ServerSocket.java:453) at java/net/ServerSocket.accept(ServerSocket.java:421) at org/apache/jk/common/ChannelSocket.accept(ChannelSocket.java:311) at org/apache/jk/common/ChannelSocket.acceptConnections(ChannelSocket.java:668) at org/apache/jk/common/ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879) at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "TP-Monitor" id=182 idx=0x2d4 tid=30211 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable@0xce48eb28[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable.run(ThreadPool.java:565) ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable@0xce48eb28[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "http-8983-Acceptor-0" id=183 idx=0x2d8 tid=30212 prio=5 alive, in native, daemon at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method) at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390) ^-- Holding lock: java/net/SocksSocketImpl@0xc67c16c0[biased lock] at java/net/ServerSocket.implAccept(ServerSocket.java:453) at java/net/ServerSocket.accept(ServerSocket.java:421) at org/apache/tomcat/util/net/DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org/apache/tomcat/util/net/JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "Thread-11" id=185 idx=0x2dc tid=30213 prio=5 alive, in native at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method) at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:390) ^-- Holding lock: java/net/SocksSocketImpl@0xcdb9c8d8[biased lock] at java/net/ServerSocket.implAccept(ServerSocket.java:453) at java/net/ServerSocket.accept(ServerSocket.java:421) at org/apache/cassandra/thrift/TCustomServerSocket.acceptImpl(TCustomServerSocket.java:117) at org/apache/cassandra/thrift/TCustomServerSocket.acceptImpl(TCustomServerSocket.java:43) at org/apache/thrift/transport/TServerTransport.accept(TServerTransport.java:31) at org/apache/cassandra/thrift/CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:106) at com/datastax/bdp/server/DseDaemon$ThriftServer.run(DseDaemon.java:340) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "pool-5-thread-1" id=186 idx=0x2e0 tid=30214 prio=5 alive, parked, native_blocked at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "attached" id=187 idx=0x2e4 tid=30024 prio=5 alive, native_waiting, daemon "pool-6-thread-1" id=189 idx=0x2ec tid=30218 prio=5 alive, parked, native_blocked -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xc8018fd8 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java/util/concurrent/ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "RMI TCP Connection(idle)" id=190 idx=0x2f0 tid=30291 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) at java/util/concurrent/SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java/util/concurrent/SynchronousQueue.poll(SynchronousQueue.java:874) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "RMI Scheduler(0)" id=191 idx=0x2f4 tid=30292 prio=5 alive, parked, native_blocked, daemon at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java/util/concurrent/DelayQueue.take(DelayQueue.java:164) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "RMI TCP Connection(2)-127.0.0.1" id=192 idx=0x2f8 tid=30296 prio=5 alive, parked, native_blocked, daemon -- Parking to wait for: java/util/concurrent/CountDownLatch$Sync@0xc21a5a90 at jrockit/vm/Locks.park0(J)V(Native Method) at jrockit/vm/Locks.park(Locks.java:2230) at sun/misc/Unsafe.park(ZJ)V(Native Method) at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158) at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:207) at org/apache/solr/core/CassandraCoreContainer.awaitShutdown(CassandraCoreContainer.java:468) at com/datastax/bdp/plugin/SolrContainerPlugin.deactivateImpl(SolrContainerPlugin.java:211) at com/datastax/bdp/plugin/PluginManager.deactivate(PluginManager.java:105) at com/datastax/bdp/plugin/PluginManager.unregister(PluginManager.java:68) ^-- Holding lock: com/datastax/bdp/plugin/PluginManager@0xc2369280[thin lock] at com/datastax/bdp/plugin/PluginManager.stopAll(PluginManager.java:204) at com/datastax/bdp/server/DseDaemon.stopServer(DseDaemon.java:260) at org/apache/cassandra/service/AbstractCassandraDaemon.stopRPCServer(AbstractCassandraDaemon.java:349) at org/apache/cassandra/service/StorageService.stopRPCServer(StorageService.java:363) at org/apache/cassandra/service/StorageService.drain(StorageService.java:2717) ^-- Holding lock: org/apache/cassandra/service/StorageService@0xc414e5e0[biased lock] at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java/lang/reflect/Method.invoke(Method.java:597) at com/sun/jmx/mbeanserver/StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com/sun/jmx/mbeanserver/StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com/sun/jmx/mbeanserver/MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com/sun/jmx/mbeanserver/PerInterface.invoke(PerInterface.java:120) at com/sun/jmx/mbeanserver/MBeanSupport.invoke(MBeanSupport.java:262) at com/sun/jmx/interceptor/DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com/sun/jmx/mbeanserver/JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax/management/remote/rmi/RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax/management/remote/rmi/RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax/management/remote/rmi/RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax/management/remote/rmi/RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax/management/remote/rmi/RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java/lang/reflect/Method.invoke(Method.java:597) at sun/rmi/server/UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun/rmi/transport/Transport$1.run(Transport.java:159) at jrockit/vm/AccessController.doPrivileged(AccessController.java:254) at sun/rmi/transport/Transport.serviceCall(Transport.java:155) at sun/rmi/transport/tcp/TCPTransport.handleMessages(TCPTransport.java:535) at sun/rmi/transport/tcp/TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun/rmi/transport/tcp/TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "JMX server connection timeout 193" id=193 idx=0x2fc tid=30297 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: [I@0xc753bee8[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at com/sun/jmx/remote/internal/ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) ^-- Lock released while waiting: [I@0xc753bee8[fat lock] at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) -- end of trace "(Attach Listener)" id=194 idx=0x300 tid=30330 prio=5 alive, daemon ===== END OF THREAD DUMP =============== sergio on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10531 Tue, 30 Apr 2013 19:50:58 +0000 sergio 10531@http://www.datastax.com/support-forums/ Hi, could you take a thread dump while it hangs and attach it here? hliu on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10529 Tue, 30 Apr 2013 18:49:44 +0000 hliu 10529@http://www.datastax.com/support-forums/ Yes, we are using solr that comes with DSE. It seems like the issue is related to "nodetool drain" operation. I don't have the hanging issue when stopping dse without doing "nodetool drain". sergio on "Support for moving Solr Indexes across DSE clusters?" http://www.datastax.com/support-forums/topic/support-for-moving-solr-indexes-across-dse-clusters#post-10506 Mon, 29 Apr 2013 15:14:09 +0000 sergio 10506@http://www.datastax.com/support-forums/ Hi, there's currently no way to move indexes, either in (that is, renaming), or between clusters: as you already noted, you'll have to rebuild it. Cheers, Sergio B. hliu on "Support for moving Solr Indexes across DSE clusters?" http://www.datastax.com/support-forums/topic/support-for-moving-solr-indexes-across-dse-clusters#post-10461 Sat, 27 Apr 2013 15:43:12 +0000 hliu 10461@http://www.datastax.com/support-forums/ Is there a way to move a Solr index from a source cluster to a target cluster without building the index at the target again? Thanks. Sven on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10370 Thu, 25 Apr 2013 21:41:38 +0000 Sven 10370@http://www.datastax.com/support-forums/ Based upon your other thread, where you using the solr that comes with DSE? How are you stopping DSE? sam on "SolrJ version for DSE 3.0.1?" http://www.datastax.com/support-forums/topic/solrj-version-for-dse-301#post-10369 Thu, 25 Apr 2013 19:50:55 +0000 sam 10369@http://www.datastax.com/support-forums/ DSE actually ships with its own version of SolrJ, which includes some extensions for Kerberos authentication. You'll find the jar, along with another containing helpers for configuring Kerberos & Solrj in the clients/ directory of the DSE installation. In package installs these are found in /usr/share/clients/ and in tarball installs in DSE_ROOT/clients/ There are also instructions on how to use the extended SolrJ with Kerberos included in the DSE distribution. In package installs this is /usr/share/dse/solr/solrj-auth-README.md and in a tarball install you'll find it at DSE_ROOT/resources/solr/solrj-auth-README.md hliu on "SolrJ version for DSE 3.0.1?" http://www.datastax.com/support-forums/topic/solrj-version-for-dse-301#post-10368 Thu, 25 Apr 2013 19:28:48 +0000 hliu 10368@http://www.datastax.com/support-forums/ The release doc of dse 3.0.1 says it depends on solr 4.0.2. I tried to find the 4.0.2 version jars of solrcore and solrj, but it seems that they are not hosted on the solr site. Could someone please give me a pointer on how to find the solrj library that fits dse 3.0.1? Thanks! hliu on "Issue with stopping DSE 3.0.1 on a single node setting" http://www.datastax.com/support-forums/topic/issue-with-stopping-dse-301-on-a-single-node-setting#post-10367 Thu, 25 Apr 2013 19:19:45 +0000 hliu 10367@http://www.datastax.com/support-forums/ I am trying to upgrade to dse3.0.1 with solr. On my dev machine, I am able to start dse3.0.1 and perform indexing on Solr. However, when I keeps on having issue with stopping dse. From the log, seems like it always hangs at the following stage: INFO [RMI TCP Connection(2)-127.0.0.1] 2013-04-25 11:54:23,865 PluginManager.java (line 104) Deactivating plugin: com.datastax.bdp.plugin.SolrContainerPlugin What might cause this issue? Thanks!