DataStax Support Forums » User Favorites: ynixon

DataStax Support Forums » User Favorites: ynixon ynixon Software, Support, and Training for Apache Cassandra en-US Sat, 18 May 2013 14:14:05 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Sven on "High MutationStage" http://www.datastax.com/support-forums/topic/high-mutationstage#post-9375 Thu, 07 Mar 2013 16:24:01 +0000 Sven 9375@http://www.datastax.com/support-forums/ What version of DSE are you using? ynixon on "High MutationStage" http://www.datastax.com/support-forums/topic/high-mutationstage#post-9374 Thu, 07 Mar 2013 16:18:21 +0000 ynixon 9374@http://www.datastax.com/support-forums/ <code> INFO [ScheduledTasks:1] 2013-03-06 15:12:21,713 GCInspector.java (line 122) GC for ParNew: 722 ms for 2 collections, 3449664536 used; max is 4125097984 INFO [OptionalTasks:1] 2013-03-06 15:12:21,943 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='company', ColumnFamily='txattr') (estimated 83552647 bytes) INFO [OptionalTasks:1] 2013-03-06 15:12:21,944 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-txattr@1400354514(27446113/83552647 serialized/live bytes, 38050 ops) INFO [ScheduledTasks:1] 2013-03-06 15:12:35,296 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1550 ms for 2 collections, 2058484368 used; max is 4125097984 INFO [FlushWriter:5] 2013-03-06 15:12:35,973 Memtable.java (line 305) Completed flushing /opt/cassandra/lib/data/company/txctx/company-txctx-hf-39-Data.db (9959461 bytes) for commitlog position ReplayPosition(segmentId=1362567378287, position=3422613) INFO [FlushWriter:5] 2013-03-06 15:12:35,975 Memtable.java (line 264) Writing Memtable-txtier@699572157(28137242/167650082 serialized/live bytes, 15474 ops) INFO [OptionalTasks:1] 2013-03-06 15:12:35,975 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='company', ColumnFamily='txwu') (estimated 83013911 bytes) INFO [ScheduledTasks:1] 2013-03-06 15:12:36,352 GCInspector.java (line 122) GC for ParNew: 417 ms for 2 collections, 2689840136 used; max is 4125097984 INFO [ScheduledTasks:1] 2013-03-06 15:12:37,356 GCInspector.java (line 122) GC for ParNew: 497 ms for 2 collections, 2581031264 used; max is 4125097984 INFO [OptionalTasks:1] 2013-03-06 15:12:38,053 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-txwu@1733440143(20321642/109106975 serialized/live bytes, 38050 ops) INFO [ScheduledTasks:1] 2013-03-06 15:12:38,610 GCInspector.java (line 122) GC for ParNew: 594 ms for 2 collections, 2723066120 used; max is 4125097984 INFO [ScheduledTasks:1] 2013-03-06 15:12:39,614 GCInspector.java (line 122) GC for ParNew: 243 ms for 1 collections, 3349962224 used; max is 4125097984 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,298 MessagingService.java (line 673) 2 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-03-06 15:12:40,301 StatusLogger.java (line 57) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2013-03-06 15:12:40,308 StatusLogger.java (line 72) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,308 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,309 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,309 StatusLogger.java (line 72) MutationStage 64 369 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,309 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,310 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,310 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,310 StatusLogger.java (line 72) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,311 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,311 StatusLogger.java (line 72) MemtablePostFlusher 1 5 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,312 StatusLogger.java (line 72) FlushWriter 1 5 1 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,312 StatusLogger.java (line 72) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,312 StatusLogger.java (line 72) commitlog_archiver 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,313 StatusLogger.java (line 72) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,313 StatusLogger.java (line 72) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,314 StatusLogger.java (line 77) CompactionManager 3 4 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,314 StatusLogger.java (line 89) MessagingService n/a 36,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,315 StatusLogger.java (line 99) Cache Type Size Capacity KeysToSave Provider INFO [ScheduledTasks:1] 2013-03-06 15:12:40,315 StatusLogger.java (line 100) KeyCache 125 2184533 all INFO [ScheduledTasks:1] 2013-03-06 15:12:40,315 StatusLogger.java (line 106) RowCache 0 0 all org.apache.cassandra.cache.SerializingCacheProvider INFO [ScheduledTasks:1] 2013-03-06 15:12:40,316 StatusLogger.java (line 113) ColumnFamily Memtable ops,data INFO [ScheduledTasks:1] 2013-03-06 15:12:40,316 StatusLogger.java (line 116) system.NodeIdInfo 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,316 StatusLogger.java (line 116) system.IndexInfo 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,317 StatusLogger.java (line 116) system.LocationInfo 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,317 StatusLogger.java (line 116) system.Versions 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,317 StatusLogger.java (line 116) system.schema_keyspaces 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,317 StatusLogger.java (line 116) system.Migrations 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,318 StatusLogger.java (line 116) system.schema_columnfamilies 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,318 StatusLogger.java (line 116) system.schema_columns 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,318 StatusLogger.java (line 116) system.HintsColumnFamily 687,76357816 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,318 StatusLogger.java (line 116) system.Schema 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,319 StatusLogger.java (line 116) OpsCenter.events_timeline 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,319 StatusLogger.java (line 116) OpsCenter.pdps 426,60560 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,319 StatusLogger.java (line 116) OpsCenter.settings 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,319 StatusLogger.java (line 116) OpsCenter.rollups7200 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,320 StatusLogger.java (line 116) OpsCenter.rollups60 1358,335766 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,320 StatusLogger.java (line 116) OpsCenter.events 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,320 StatusLogger.java (line 116) OpsCenter.rollups300 428,142167 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,320 StatusLogger.java (line 116) OpsCenter.rollups86400 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,321 StatusLogger.java (line 116) company.txmetric 49050,35996029 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,321 StatusLogger.java (line 116) company.txattr 9500,30080972 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,321 StatusLogger.java (line 116) company.txctx 13500,40387725 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,321 StatusLogger.java (line 116) company.txtier 47550,84303564 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,322 StatusLogger.java (line 116) company.objects 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,322 StatusLogger.java (line 116) company.txwu 0,0 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,322 StatusLogger.java (line 116) company.periods 360,10119 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,322 StatusLogger.java (line 116) company.attributes 900,5820 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,323 StatusLogger.java (line 116) company.tiers 9924,86081 INFO [ScheduledTasks:1] 2013-03-06 15:12:40,323 StatusLogger.java (line 116) company.applications 122,3112 INFO [FlushWriter:5] 2013-03-06 15:12:40,716 Memtable.java (line 305) Completed flushing /opt/cassandra/lib/data/company/txtier/company-txtier-hf-23-Data.db (8991230 bytes) for commitlog position ReplayPosition(segmentId=1362567378287, position=3422613) INFO [FlushWriter:5] 2013-03-06 15:12:40,718 Memtable.java (line 264) Writing Memtable-HintsColumnFamily@908436622(111743540/139679425 serialized/live bytes, 1472 ops) INFO [ScheduledTasks:1] 2013-03-06 15:12:42,752 GCInspector.java (line 122) GC for ParNew: 437 ms for 2 collections, 3344201360 used; max is 4125097984 INFO [OptionalTasks:1] 2013-03-06 15:12:42,812 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='company', ColumnFamily='txtier') (estimated 107011538 bytes) INFO [MemoryMeter:1] 2013-03-06 15:12:42,911 Memtable.java (line 213) CFS(Keyspace='company', ColumnFamily='txmetric') liveRatio is 5.817172512218668 (just-counted was 4.962854126542127). calculation took 28767ms for 104046 columns INFO [OptionalTasks:1] 2013-03-06 15:12:43,403 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-txtier@1666775274(18083059/107744260 serialized/live bytes, 64050 ops) INFO [CompactionExecutor:84] 2013-03-06 15:12:43,458 CompactionTask.java (line 221) Compacted to [/opt/cassandra/lib/data/company/txwu/company-txwu-hf-36-Data.db,/opt/cassandra/lib/data/company/txwu/company-txwu-hf-39-Data.db,]. 62,913,763 to 62,910,862 (~99% of original) bytes for 2 keys at 0.990989MB/s. Time: 60,542ms. INFO [CompactionExecutor:84] 2013-03-06 15:12:43,478 CompactionTask.java (line 109) Compacting [SSTableReader(path='/opt/cassandra/lib/data/company/txwu/company-txwu-hf-37-Data.db'), SSTableReader(path='/opt/cassandra/lib/data/company/txwu/company-txwu-hf-38-Data.db'), SSTableReader(path='/opt/cassandra/lib/data/company/txwu/company-txwu-hf-36-Data.db'), SSTableReader(path='/opt/cassandra/lib/data/company/txwu/company-txwu-hf-40-Data.db')] INFO [CompactionExecutor:84] 2013-03-06 15:12:43,482 CompactionController.java (line 172) Compacting large row company/txwu:2013-03-6-13:10-wu (169764814 bytes) incrementally INFO [ScheduledTasks:1] 2013-03-06 15:12:57,975 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1693 ms for 2 collections, 2375045832 used; max is 4125097984 </code> ynixon on "High MutationStage" http://www.datastax.com/support-forums/topic/high-mutationstage#post-9373 Thu, 07 Mar 2013 16:15:33 +0000 ynixon 9373@http://www.datastax.com/support-forums/ 2 node cluster, Machines with 16GB, redhat 5 64Bit, 8 cores network topology , CF 3 5 Column familes with 25 writes per second each, zero reads on load test (written in Hector) we get some MutationStage Pending in system.log / nodetool tpstats hitting this errors every 2-10 minutes At the very same time of high MutationStage (pending) the client is getting also timeout We also noticed that it keeps running large minor compaction - that consume alot of time which limit we reached: network/IO/number of nodes/configuration ? Srini on "cassandra health check" http://www.datastax.com/support-forums/topic/cassandra-health-check#post-8013 Tue, 18 Dec 2012 17:47:53 +0000 Srini 8013@http://www.datastax.com/support-forums/ These metrics should be helpful Read/Write Count and read/Write Latency This link has detailed doc on various JMX metrics <a href="http://www.datastax.com/docs/0.8/operations/monitoring" rel="nofollow">http://www.datastax.com/docs/0.8/operations/monitoring</a> nickmbailey on "recommended JMX metrics to monitor health check" http://www.datastax.com/support-forums/topic/recommended-jmx-metrics-to-monitor-health-check#post-8011 Tue, 18 Dec 2012 16:30:42 +0000 nickmbailey 8011@http://www.datastax.com/support-forums/ For starters I would recommend browsing the StorageProxyMBean and the StorageServiceMBean. Additionally there are per column family mbeans with more specific information. ynixon on "recommended JMX metrics to monitor health check" http://www.datastax.com/support-forums/topic/recommended-jmx-metrics-to-monitor-health-check#post-8010 Tue, 18 Dec 2012 16:08:32 +0000 ynixon 8010@http://www.datastax.com/support-forums/ We have JMX based monitor and want to define some necessary metrics Which metrics/MBeans are recommended ? nickmbailey on "omitting console output at Cassandra startup" http://www.datastax.com/support-forums/topic/omitting-console-output-at-cassandra-startup#post-8009 Tue, 18 Dec 2012 15:48:09 +0000 nickmbailey 8009@http://www.datastax.com/support-forums/ I believe you mean opscenter-agent startup? Unfortunately those messages can't be removed at the moment. You could redirect standard out to a file or /dev/null though. ynixon on "cassandra health check" http://www.datastax.com/support-forums/topic/cassandra-health-check#post-8005 Tue, 18 Dec 2012 10:12:20 +0000 ynixon 8005@http://www.datastax.com/support-forums/ OpsCenter is good if you have no other central monitor. We have JMX based monitor Which metrics/MBeans are recommended ? ynixon on "omitting console output at Cassandra startup" http://www.datastax.com/support-forums/topic/omitting-console-output-at-cassandra-startup#post-8004 Tue, 18 Dec 2012 08:54:37 +0000 ynixon 8004@http://www.datastax.com/support-forums/ after cassandra startup I am getting at the console: $ log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log). log4j:WARN Please initialize the log4j system properly. log4j:WARN See <a href="http://logging.apache.org/log4j/1.2/faq.html#noconfig" rel="nofollow">http://logging.apache.org/log4j/1.2/faq.html#noconfig</a> for more info. INFO [main] 2012-12-18 10:49:45,773 Loading conf files: ./conf/address.yaml INFO [main] 2012-12-18 10:49:45,825 Java vendor/version: OpenJDK 64-Bit Server VM/1.6.0_22 INFO [main] 2012-12-18 10:49:45,825 Waiting for the config from OpsCenter INFO [main] 2012-12-18 10:49:45,829 SSL communication is disabled INFO [main] 2012-12-18 10:49:45,829 Creating stomp connection to 172.17.4.26:61620 INFO [main] 2012-12-18 10:49:45,837 Attempting to determine Cassandra's broadcast address through JMX INFO [StompConnection receiver] 2012-12-18 10:49:45,838 Reconnecting in 0s. how can I omit these console output (stdout) messages ? (I tried and tried to figure out log4j-server.properties, with no success) ynixon on "nodetool immediate after bin/cassandra --> Connection refused" http://www.datastax.com/support-forums/topic/nodetool-immediate-after-bincassandra-connection-refused#post-8003 Tue, 18 Dec 2012 08:46:30 +0000 ynixon 8003@http://www.datastax.com/support-forums/ thanks, used your suggestion and it is working great nickmbailey on "nodetool immediate after bin/cassandra --> Connection refused" http://www.datastax.com/support-forums/topic/nodetool-immediate-after-bincassandra-connection-refused#post-7991 Mon, 17 Dec 2012 16:27:55 +0000 nickmbailey 7991@http://www.datastax.com/support-forums/ The problem is that JMX doesn't come up immediately when you start the jvm. It shouldn't take too long but a better way might be to check nodetool in a loop until JMX is listening. ynixon on "nodetool immediate after bin/cassandra --> Connection refused" http://www.datastax.com/support-forums/topic/nodetool-immediate-after-bincassandra-connection-refused#post-7990 Mon, 17 Dec 2012 16:16:56 +0000 ynixon 7990@http://www.datastax.com/support-forums/ I wrote a script that start cassandra and then check its status this way I have erorr: $ bin/cassandra -p cassandra.pid > /dev/null 2>/dev/null ; bin/nodetool ring Failed to connect to '127.0.0.1:7199': Connection refused If I add sleep 10 between bin/cassandra & bin/nodetool - it work fine my concern is that in a busy system, 10 sec will not be enough .... Isn't any smarter way for this ? ynixon on "failed: Connection was refused by other side: 111: Connection refused." http://www.datastax.com/support-forums/topic/failed-connection-was-refused-by-other-side-111-connection-refused#post-7989 Mon, 17 Dec 2012 16:05:28 +0000 ynixon 7989@http://www.datastax.com/support-forums/ thanks, you information helped mbulman on "failed: Connection was refused by other side: 111: Connection refused." http://www.datastax.com/support-forums/topic/failed-connection-was-refused-by-other-side-111-connection-refused#post-7986 Mon, 17 Dec 2012 09:28:42 +0000 mbulman 7986@http://www.datastax.com/support-forums/ ynixon, opscenterd will only create agent.tar.gz if ssl is enabled (it is by default), and setup.py has not already been run for that instance. It should be something that you only need to run once. You can look in opscenterd.log for messages starting with "Configuring agent communication". That will let you know if SSL is enabled or not. If it is, and bin/setup.py is going to run, you'll see another log message that says "SSL files do not exist. Attempting to create them now....". ynixon on "failed: Connection was refused by other side: 111: Connection refused." http://www.datastax.com/support-forums/topic/failed-connection-was-refused-by-other-side-111-connection-refused#post-7985 Mon, 17 Dec 2012 09:18:59 +0000 ynixon 7985@http://www.datastax.com/support-forums/ From the documentation I understand that the first time opscenter is started, a file called agent.tar.gz should create automatically at some former installations it happened and somehow it is not happening anymore :( After some reading I understand that this file can be created by running bin/setup.py should I relay on the automatic way or manual way ?