Hello
I am having 3 cassandra servers running in cluster.
root@cassandra3:/opt/apache/apache-cassandra-0.8.10# ./bin/nodetool -h 127.0.0.1 ring
Address DC Rack Status State Load Owns Token
113427455640312821154458202477256070485
172.19.10.1 19 10 Up Normal 166.6 MB 33.33% 0
172.19.10.2 19 10 Up Normal 166.71 MB 33.33% 56713727820156410577229101238628035242
172.19.10.3 19 10 Up Normal 166.8 MB 33.33% 113427455640312821154458202477256070485
As you see all of them are in normal mode and up.
Most of operations are working except adding a counter which fails ( not always and not for single column family ).
In cassandra server.log i dont see any error at all.
On client side i get timeout exception:
2013-01-06 15:37:01,152 ERROR [javax.slee.SbbNotification[service=ServiceID[name=Online Billing,vendor=Web Ukraine,version=1.0.0.0],sbb=SbbID[name=Online Billing SBB,vendor=Web Ukraine,version=1.0.0.0]].OnlineBillingSbb] ERROR OCCURED
TimedOutException()
at org.apache.cassandra.thrift.Cassandra$add_result.read(Cassandra.java:16336)
at org.apache.cassandra.thrift.Cassandra$Client.recv_add(Cassandra.java:903)
at org.apache.cassandra.thrift.Cassandra$Client.add(Cassandra.java:875)
at com.webukraine.ims.billing.online.db.CassandraController.add(Unknown Source)
at com.webukraine.ims.billing.online.db.BillingCyclesManager.updateCycleBalance(Unknown Source)
at com.webukraine.ims.billing.online.db.ResourceAllocator.allocateUnits(Unknown Source)
at com.webukraine.ims.billing.online.OnlineBillingSbb.onRoCreditControlRequest(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.invokeEventHandler(SbbEntityImpl.java:478)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:378)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:126)
at org.mobicents.slee.runtime.eventrouter.EventRouterExecutorImpl$EventRoutingTaskStatsCollector.run(EventRouterExecutorImpl.java:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Another example :
2013-01-06 15:59:39,331 ERROR [javax.slee.SbbNotification[service=ServiceID[name=Online Billing,vendor=Web Ukraine,version=1.0.0.0],sbb=SbbID[name=Online Billing SBB,vendor=Web Ukraine,version=1.0.0.0]].OnlineBillingSbb] ERROR OCCURED
TimedOutException()
at org.apache.cassandra.thrift.Cassandra$add_result.read(Cassandra.java:16336)
at org.apache.cassandra.thrift.Cassandra$Client.recv_add(Cassandra.java:903)
at org.apache.cassandra.thrift.Cassandra$Client.add(Cassandra.java:875)
at com.webukraine.ims.billing.online.db.CassandraController.add(Unknown Source)
at com.webukraine.ims.billing.online.db.BillingCyclesManager.updateUnitsLeft(Unknown Source)
at com.webukraine.ims.billing.online.db.ResourceAllocator.allocateUnits(Unknown Source)
at com.webukraine.ims.billing.online.OnlineBillingSbb.onRoCreditControlRequest(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mobicents.slee.runtime.sbbentity.SbbEntityImpl.invokeEventHandler(SbbEntityImpl.java:478)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.routeQueuedEvent(EventRoutingTaskImpl.java:378)
at org.mobicents.slee.runtime.eventrouter.routingtask.EventRoutingTaskImpl.run(EventRoutingTaskImpl.java:126)
at org.mobicents.slee.runtime.eventrouter.EventRouterExecutorImpl$EventRoutingTaskStatsCollector.run(EventRouterExecutorImpl.java:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
After restarting cassandra servers everything is working as expected but this is not good thing to restart a database.
How this problem can be resolved?
Best regards
Yulian Oifa