interpreting system.log

(7 posts) (2 voices)

• Started 6 months ago by matteraf
• Latest reply from nickmbailey
• This topic is not a support question

Tags:

No tags yet.

1. 

   matteraf
   Member

   Hi,

   We have C* monitored by Nagios, yesterday this message come to my email box:

   Notification Type: PROBLEM

   Service: Row read stage
   Host: xxxxxxxxxxxx
   Address: 1.9.7.8.0
   State: WARNING

   Date/Time: Mon Dec 10 19:35:03 EST 2012

   Additional Info:
   JMX WARNING PendingTasks.used=2

   and this is the output from system log at this time:

   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,917 MessagingService.java (line 612) 1 READ messages dropped in server lifetime
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,917 StatusLogger.java (line 50) Pool Name Active Pending Blocked
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReadStage 32 227 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) RequestResponseStage 2 8 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReadRepairStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) MutationStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,918 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) GossipStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) AntiEntropyStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) MigrationStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,919 StatusLogger.java (line 65) StreamStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FILEUTILS-DELETE-POOL 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FlushWriter 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) MiscStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,920 StatusLogger.java (line 65) FlushSorter 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,921 StatusLogger.java (line 65) InternalResponseStage 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 65) HintedHandoff 0 0 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 69) CompactionManager n/a 0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,933 StatusLogger.java (line 81) MessagingService n/a 0,0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 85) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.NodeIdInfo 0,0 0/0 1/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.IndexInfo 0,0 0/0 0/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.LocationInfo 0,0 0/0 1/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.Migrations 0,0 0/0 0/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,934 StatusLogger.java (line 88) system.HintsColumnFamily 0,0 0/0 1/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) system.Schema 0,0 0/0 1/1
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) FSM_TRACKING.ConversionPerAirlineKeys 0,0 0/0 0/0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) FSM_TRACKING.ConversionPerAirline 0,0 0/0 0/0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.settings 0,0 0/0 0/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.events_timeline 0,0 0/0 3/5
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,935 StatusLogger.java (line 88) OpsCenter.rollups7200 0,0 0/0 50/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,936 StatusLogger.java (line 88) OpsCenter.rollups86400 0,0 0/0 0/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.events 0,0 0/0 50/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.pdps 942,312080 0/0 300/300
   INFO [ScheduledTasks:1] 2012-12-10 19:33:12,999 StatusLogger.java (line 88) OpsCenter.rollups60 5631,7223004 0/0 50/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) OpsCenter.rollups300 3292,7400109 0/0 50/50
   INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.TopSearched 0,0 0/0 0/0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.TopSearchedByCountry 0,0 0/0 0/0
   INFO [ScheduledTasks:1] 2012-12-10 19:33:13,000 StatusLogger.java (line 88) FSM_BEST_PRICES.FlightBestPrices24Hours 53082,142064584 0/0 200000/200000

   The question is, when I have this trouble, what thing must I see?

   Posted 6 months ago #
2. 

   nickmbailey
   Key Master

   That alert appears to be from nagios hitting some JMX call, not from reading the log.

   Posted 6 months ago #
3. 

   matteraf
   Member

   Yes Nick, Nagios report pending tasks, hence I want to see something in system.log concerning about this.
   And I found these bunch of messages at the same time.

   Posted 6 months ago #
4. 

   nickmbailey
   Key Master

   Well that log is showing the threadpool counts for lots of different threadpools within cassandra, but the alert doesn't seem to indicate which threadpool it was checking pending tasks on and alerting for.

   Posted 6 months ago #
5. 

   matteraf
   Member

   what about "RequestResponseStage"?

   Posted 6 months ago #
6. 

   matteraf
   Member

   Nic, I'm a former Oracle DBA and I'm used to see alert.log to find any issues, or read dynamic performance views. So, in C* I have system.log. What can I see in that in order to find any trouble?

   Posted 6 months ago #
7. 

   nickmbailey
   Key Master

   A good thing to look for would be anything logged at WARN/ERROR/FATAL.

   As you've noticed, threadpool information is periodically logged but I would recommend setting up alerts for these values by checking jmx rather than examining the log. A pending value of 2 is likely not worrisome. If a threadpool has pending tasks start to grow and get backed up, that would be worth alerting on.

   Posted 6 months ago #

RSS feed for this topic