DataStax Support Forums » Topic: interpreting system.log

DataStax Support Forums » Topic: interpreting system.log http://www.datastax.com/support-forums/topic/interpreting-systemlog Software, Support, and Training for Apache Cassandra en-US Thu, 20 Jun 2013 07:22:59 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php nickmbailey on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7955 Tue, 11 Dec 2012 18:13:14 +0000 nickmbailey 7955@http://www.datastax.com/support-forums/ A good thing to look for would be anything logged at WARN/ERROR/FATAL. As you've noticed, threadpool information is periodically logged but I would recommend setting up alerts for these values by checking jmx rather than examining the log. A pending value of 2 is likely not worrisome. If a threadpool has pending tasks start to grow and get backed up, that would be worth alerting on. matteraf on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7954 Tue, 11 Dec 2012 17:45:24 +0000 matteraf 7954@http://www.datastax.com/support-forums/ Nic, I'm a former Oracle DBA and I'm used to see alert.log to find any issues, or read dynamic performance views. So, in C* I have system.log. What can I see in that in order to find any trouble? matteraf on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7953 Tue, 11 Dec 2012 17:42:01 +0000 matteraf 7953@http://www.datastax.com/support-forums/ what about "RequestResponseStage"? nickmbailey on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7951 Tue, 11 Dec 2012 16:58:56 +0000 nickmbailey 7951@http://www.datastax.com/support-forums/ Well that log is showing the threadpool counts for lots of different threadpools within cassandra, but the alert doesn't seem to indicate which threadpool it was checking pending tasks on and alerting for. matteraf on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7950 Tue, 11 Dec 2012 16:56:55 +0000 matteraf 7950@http://www.datastax.com/support-forums/ Yes Nick, Nagios report pending tasks, hence I want to see something in system.log concerning about this. And I found these bunch of messages at the same time. nickmbailey on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7945 Tue, 11 Dec 2012 16:26:45 +0000 nickmbailey 7945@http://www.datastax.com/support-forums/ That alert appears to be from nagios hitting some JMX call, not from reading the log. matteraf on "interpreting system.log" http://www.datastax.com/support-forums/topic/interpreting-systemlog#post-7941 Tue, 11 Dec 2012 13:57:10 +0000 matteraf 7941@http://www.datastax.com/support-forums/ Hi, We have C* monitored by Nagios, yesterday this message come to my email box: Notification Type: PROBLEM Service: Row read stage Host: xxxxxxxxxxxx Address: 1.9.7.8.0 State: WARNING Date/Time: Mon Dec 10 19:35:03 EST 2012 Additional Info: JMX WARNING PendingTasks.used=2 and this is the output from system log at this time: INFO [ScheduledTasks:1] 2012-12-10 19:33:12,917 MessagingService.java (line 612) 1 READ messages dropped in server lifetime INFO [ScheduledTasks:1] 2012-12-10 19:33:12,917 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReadStage 32 227 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) RequestResponseStage 2 8 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) MutationStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FILEUTILS-DELETE-POOL 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FlushWriter 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FlushSorter 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,921 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 69) CompactionManager n/a 0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 81) MessagingService n/a 0,0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 85) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.NodeIdInfo 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.IndexInfo 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.LocationInfo 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.Migrations 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.HintsColumnFamily 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) system.Schema 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) FSM_TRACKING.ConversionPerAirlineKeys 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) FSM_TRACKING.ConversionPerAirline 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.settings 0,0 0/0 0/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.events_timeline 0,0 0/0 3/5 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.rollups7200 0,0 0/0 50/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,936 StatusLogger.java (line 88) OpsCenter.rollups86400 0,0 0/0 0/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.events 0,0 0/0 50/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.pdps 942,312080 0/0 300/300 INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.rollups60 5631,7223004 0/0 50/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) OpsCenter.rollups300 3292,7400109 0/0 50/50 INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.TopSearched 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.TopSearchedByCountry 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.FlightBestPrices24Hours 53082,142064584 0/0 200000/200000 The question is, when I have this trouble, what thing must I see?