OpsCenter Agent Exception

(12 posts) (2 voices)

• Started 6 months ago by GodenYao
• Latest reply from GodenYao
• This topic is not resolved

Tags:

No tags yet.

1. 

   GodenYao
   Member

   this is a follow-up question. the root cause of my OpsCenter Agent went no data (but still connected) is what I found in the log below:
   How can I prevent this from happening again and again?

   WARN [Thread-3] 2012-11-21 01:11:29,330 Could not fullfill request on this host CassandraClient<10.2.255.1:9160-16>
   WARN [Thread-3] 2012-11-21 01:11:29,330 Exception:
   me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
   at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
   at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:252)
   at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
   at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
   at clj_hector.core$put.doInvoke(core.clj:164)
   at clojure.lang.RestFn.invoke(RestFn.java:470)
   at opsagent.cassandra$store_rollup.invoke(cassandra.clj:107)
   at clojure.lang.AFn.applyToHelper(AFn.java:161)
   at clojure.lang.AFn.applyTo(AFn.java:151)
   at clojure.core$apply.invoke(core.clj:540)
   at opsagent.cassandra$async_call$fn__582$fn__583.invoke(cassandra.clj:164)
   at opsagent.cassandra$process_queue$fn__587.invoke(cassandra.clj:170)
   at opsagent.cassandra$process_queue.invoke(cassandra.clj:169)
   at opsagent.cassandra$setup_cassandra$fn__595.invoke(cassandra.clj:203)
   at clojure.lang.AFn.run(AFn.java:24)
   at java.lang.Thread.run(Thread.java:722)
   Caused by: TimedOutException()
   at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19493)
   at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
   at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
   at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
   at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
   at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
   at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:246)
   ... 14 more
   INFO [qtp176048939-3883] 2012-11-21 01:11:40,764 HTTP: :get /os-metric/disk-space - 200
   INFO [StompConnection receiver] 2012-11-21 01:11:40,771 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :thrift_rpc_interface "10.2.255.1", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter", :rollup_subscriptions []}

   Posted 6 months ago #
2. 

   nickmbailey
   Key Master

   Are there many of those exceptions? That exception indicates that Cassandra timed out while OpsCenter was doing an insert. The OpsCenter agent will retry the insert however.

   Posted 6 months ago #
3. 

   GodenYao
   Member

   it's kinda frequent - maybe it's related to our recent schema change. I'll keep watching.

   Posted 6 months ago #
4. 

   GodenYao
   Member

   When you said it'll retry - when will that happen - from the log i don't see it retry and that's when I noticed I don't have data from the nodes.

   The only thing to bring it back is to restart agent service.

   Posted 6 months ago #
5. 

   nickmbailey
   Key Master

   When the agent encounters an error like that it will put the operation back in its queue of writes and then wait 5 seconds before starting to process inserts again. You won't see any special logging for the retry however.

   There could be another issue with your agents that is causing them to stop writing data. I would need to inspect the logs to be sure though.

   Posted 6 months ago #
6. 

   GodenYao
   Member

   after i restarted all agents, some are back to inserting data. some are not.
   here's the problematic one's log:
   It seems that it has some issues and keep waiting...
   Any extra log you need for diagnosing?

   Starting OpsCenter agent monitor opscenter_agent_monitor.
   INFO [main] 2012-11-21 05:54:59,259 Loading conf files: /var/lib/opscenter-agent/conf/address.yaml
   INFO [main] 2012-11-21 05:54:59,336 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_05
   INFO [main] 2012-11-21 05:54:59,336 Waiting for the config from OpsCenter
   INFO [main] 2012-11-21 05:54:59,337 SSL communication is disabled
   INFO [main] 2012-11-21 05:54:59,337 Creating stomp connection to 10.2.255.6:61620
   INFO [StompConnection receiver] 2012-11-21 05:54:59,340 Reconnecting in 0s.
   INFO [main] 2012-11-21 05:54:59,341 Attempting to determine Cassandra's broadcast address through JMX
   INFO [StompConnection receiver] 2012-11-21 05:54:59,345 Connected to 10.2.255.6:61620
   INFO [Initialization] 2012-11-21 05:54:59,348 Sleeping for 2s before trying to determine IP over JMX again
   INFO [StompConnection receiver] 2012-11-21 05:54:59,414 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"}
   INFO [StompConnection receiver] 2012-11-21 05:54:59,420 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"}
   INFO [Initialization] 2012-11-21 05:55:01,350 New JMX connection (127.0.0.1:7199)
   INFO [Initialization] 2012-11-21 05:55:01,504 Sleeping for 4s before trying to determine IP over JMX again
   INFO [StompConnection receiver] 2012-11-21 05:55:01,554 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"}
   INFO [Initialization] 2012-11-21 05:55:05,505 New JMX connection (127.0.0.1:7199)
   INFO [Initialization] 2012-11-21 05:55:05,515 Sleeping for 8s before trying to determine IP over JMX again

   Posted 6 months ago #
7. 

   nickmbailey
   Key Master

   That log is indicating that the agent is having a problem connecting to cassandra and determining its ip address. A couple things could cause that. Can you change the agent log level to DEBUG and attach that log, as well as attaching the log from the main opscenter machine for a few minutes after you restart the agent.

   Posted 6 months ago #
8. 

   GodenYao
   Member

   i restarted my opscenter service (on node .6) and the problematic one (on node .1).
   now dashboard shows .1 is disconnected from agent perspective.

   opsCenter agent log on .1 is pretty much similar to what I posted.
   opsCenter log on .6 is below (enabled DEBUG)

   ==== Log ==== wonder if there's a way to attach a file instead of pasting everything here.

   2012-11-21 06:35:29+0800 [Weibook] INFO: Starting services for cluster Weibook
   2012-11-21 06:35:29+0800 [] DEBUG: Getting nodelist over thrift
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.7:9160 @0x3035200>
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.6:9160 @0x3035440>
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.5:9160 @0x3035248>
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.4:9160 @0x30352d8>
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.3:9160 @0x318c518>
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Waiting for describe_version() results
   2012-11-21 06:35:29+0800 [Weibook] INFO: Starting CassandraCluster service
   2012-11-21 06:35:29+0800 [Weibook] INFO: agent_config items: {'rollups60_ttl': 604800, 'thrift_port': 9160, 'rollups7200_ttl': 31536000, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'}
   2012-11-21 06:35:29+0800 [Weibook] INFO: Starting PushService
   2012-11-21 06:35:29+0800 [] INFO: OS Version: Linux version 2.6.32-5-amd64 (Debian 2.6.32-46) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Sun Sep 23 10:07:46 UTC 2012
   2012-11-21 06:35:29+0800 [] INFO: CPU Info: ['2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250']
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.6:9160 @0x3035440> to the pool
   2012-11-21 06:35:29+0800 [] INFO: Mem Info: 96814MB
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.5:9160 @0x3035248> to the pool
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Getting nodelist over thrift
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.3:9160 @0x318c518> to the pool
   2012-11-21 06:35:29+0800 [Weibook] INFO: Partitioner: org.apache.cassandra.dht.RandomPartitioner
   2012-11-21 06:35:29+0800 [Weibook] INFO: Cluster Name: Weibook
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.6 ('106338239662793269832304564822427566080')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.0 ('0')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.3 ('127605887595351923798765477786913079296')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.7 ('148873535527910577765226390751398592512')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.4 ('21267647932558653966460912964485513216')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.2 ('85070591730234615865843651857942052864')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.1 ('42535295865117307932921825928971026432')
   2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.5 ('63802943797675961899382738893456539648')
   2012-11-21 06:35:29+0800 [Weibook] DEBUG: Getting nodelist over thrift
   2012-11-21 06:35:30+0800 [Weibook] INFO: Keyspaces: [CassandraKeyspace(name=Tortoise, column_families=[], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=TestKeyspace, column_families=['Test1'], attributes={'strategy_options': {'datacenter1': '1', 'replication_factor': None}, 'replica_placement_strategy': 'org.apache.cassandra.locator.NetworkTopologyStrategy'}), CassandraKeyspace(name=system, column_families=['schema_columnfamilies', 'NodeIdInfo', 'schema_columns', 'Versions', 'IndexInfo', 'Schema', 'schema_keyspaces', 'Migrations', 'LocationInfo', 'HintsColumnFamily'], attributes={'strategy_options': {'replication_factor': None}, 'replica_placement_strategy': 'org.apache.cassandra.locator.LocalStrategy'}), CassandraKeyspace(name=SinaMiniBlog, column_families=['User', 'BlogLog', 'MidUid', 'User2', 'Alert5', 'EditoridUid', 'Alert6', 'BlogForwarded', 'FollowerLog', 'Mark', 'MentioningInBlog', 'Replying', 'LinkInComment', 'UserFollowRank', 'UserTagByEditor', 'Blog', 'Replied', 'CommentLog', 'Forwarding', 'UserUrlUid', 'Comment', 'CidMid', 'Commented', 'UserLog', 'CrawlerData', 'Follower', 'MentioningInComment', 'MentionedInBlog', 'FollowLink', 'FollowingLog', 'CrawlerDataInfo', 'GlobalConfiguration', 'Commenting', 'LinkInBlog', 'Forwarded', 'UserNameUid', 'MentionedInComment'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=TencentMiniBlog, column_families=[], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=Weibook, column_families=['UserNameToUid', 'Exception', 'Machine', 'AccountEvent2', 'ParentTagTag', 'TortoiseDomainUrl', 'Report', 'Account', 'ExceptionLog', 'BotAccount', 'TortoiseResponse', 'CrawlerAssemblySignature2', 'RealNameInfo', 'AccountEvent', 'KeywordBag', 'CrawlerAssembly', 'TortoiseTarget2', 'Tag', 'TortoiseTarget', 'TagTypeTag'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=OpsCenter, column_families=['pdps', 'rollups86400', 'rollups7200', 'events', 'rollups300', 'events_timeline', 'settings', 'rollups60'], attributes={'strategy_options': {'replication_factor': '2'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=Viizii, column_families=['User', 'UserReadHistory2', 'InboxByViz', 'Viz', 'DevicesMap', 'Inbox', 'UserFavorite'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'})]
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Current number of column families: 83
   2012-11-21 06:35:30+0800 [] /usr/share/opscenter/lib/py/orbited/cometsession.py:467: exceptions.DeprecationWarning: twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource.
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.7:9160 @0x3035200> to the pool
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Getting nodelist over thrift
   2012-11-21 06:35:30+0800 [Weibook] INFO: Initializing event storage.
   2012-11-21 06:35:30+0800 [Weibook] INFO: Attempting to load all persisted alert rules
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Attempting to load all persisted scheduled job descriptions
   2012-11-21 06:35:30+0800 [Weibook] INFO: Done loading persisted scheduled job descriptions
   2012-11-21 06:35:30+0800 [Weibook] INFO: Done loading persisted alert rules
   2012-11-21 06:35:30+0800 [Weibook] INFO: Done initializing event storage.
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.4:9160 @0x30352d8> to the pool
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1668265263/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1758260838/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'agent_rpc_broadcast_address': u'10.2.255.0', u'tasks': {u'Repair of SinaMiniBlog.Follower': 0.39552159999999997}, u'other-devices': [u'sde', u'sdb', u'sdf', u'sdd', u'sdc'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'mars', u'network-interfaces': [u'lo', u'eth0', u'eth1', u'eth2'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.4': 0.72576505000000002, u'10.2.255.7': 0.060971785000000001}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3747670659989.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 1.6399999999999999, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.0 as the RPC address for node 10.2.255.0
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node mars is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Version: {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.0 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /987724241/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'agent_rpc_broadcast_address': u'10.2.255.7', u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.24895886, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.52155289999999999, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0430843999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.88131904999999999, u'Compaction of SinaMiniBlog.Blog': 0.86973643, u'Compaction of SinaMiniBlog.UserNameUid': 0.19911227000000001}, u'other-devices': [u'sdd', u'sdb', u'sdc', u'sde'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'uranus', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1191072487259.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 17.57, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.7 as the RPC address for node 10.2.255.7
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node uranus is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.7 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /37324024/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'agent_rpc_broadcast_address': u'10.2.255.2', u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28101482999999999, u'Repair of SinaMiniBlog.Blog': 0.31844539999999999}, u'other-devices': [u'sdd', u'sdf', u'sde', u'sdc', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'venus', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.1', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.6': 0.34552179999999999}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3300647361169.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.2800000000000002, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.2 as the RPC address for node 10.2.255.2
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node venus is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.2 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Node reported as being down: 10.2.255.1
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /214866319/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'agent_rpc_broadcast_address': u'10.2.255.6', u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.18080822999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.52787859999999998, u'Compaction of SinaMiniBlog.UserNameUid': 0.039741400000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.6784519}, u'other-devices': [u'sde', u'sdb', u'sdc', u'sdd'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'saturn', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1172552961223.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 10.74, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.6 as the RPC address for node 10.2.255.6
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node saturn is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.6 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1501295681/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'agent_rpc_broadcast_address': u'10.2.255.3', u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.7930199, u'Compaction of SinaMiniBlog.BlogLog': 0.93576820000000005}, u'other-devices': [u'sde', u'sdc', u'sdf', u'sdd', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'mercury', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.1', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.7': 0.19626610999999999, u'10.2.255.6': 0.34316286000000001}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3228950748481.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 2.2000000000000002, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.3 as the RPC address for node 10.2.255.3
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node mercury is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.3 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /774784601/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'agent_rpc_broadcast_address': u'10.2.255.4', u'tasks': {u'Repair of SinaMiniBlog.UserNameUid': 0.33712560000000003}, u'other-devices': [u'sdb', u'sdc', u'sdd', u'sde'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'jupiter', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1535942583196.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.75, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.4 as the RPC address for node 10.2.255.4
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node jupiter is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.4 changed its mode to normal
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1739664394/conf
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'agent_rpc_broadcast_address': u'10.2.255.5', u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11971977}, u'other-devices': [u'sdd', u'sde', u'sdc', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'earth', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 2266018552730.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.3899999999999997, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.5 as the RPC address for node 10.2.255.5
   2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node earth is {'cassandra': '1.1.6'}
   2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.5 changed its mode to normal
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3399999999999999}
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1800000000000002}
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5900000000000001}
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.449999999999999}
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 10.359999999999999}
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996}
   2012-11-21 06:35:33+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:33+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.4500000000000002}
   2012-11-21 06:35:35+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:35+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747670669100.0, u'streams': {u'10.2.255.4': 0.73279110000000003, u'10.2.255.7': 0.062090915000000003}}
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.79573579999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.93895154999999997}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.34717967999999999}}
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3899999999999997}
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1699999999999999}
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.6200000000000001}
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.170000000000002}
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 10.09}
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.2477385, u'Compaction of SinaMiniBlog.Forwarding': 0.55636202999999995, u'Compaction of SinaMiniBlog.UserNameUid': 0.039741400000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.68375010000000003}}
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'data-held': 2266018575953.0, u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11977488}}
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3300000000000001}
   2012-11-21 06:35:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:39+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.62656290000000003, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.53907627000000002, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.054003, u'Compaction of SinaMiniBlog.BlogLog': 0.88546765000000005, u'Compaction of SinaMiniBlog.Blog': 0.88998169999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.19911227000000001}}
   2012-11-21 06:35:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:39+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.3300000000000001}
   2012-11-21 06:35:40+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:40+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28108576000000002, u'Repair of SinaMiniBlog.Blog': 0.31848925}, u'streams': {u'10.2.255.6': 0.34963095}}
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2400000000000002}
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6799999999999997}
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5700000000000001}
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.16}
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.7599999999999998}
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3099999999999996}
   2012-11-21 06:35:44+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:44+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.3099999999999996}
   2012-11-21 06:35:45+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:45+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.73832750000000003, u'10.2.255.7': 0.063278249999999994}}
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.79805479999999995, u'Compaction of SinaMiniBlog.BlogLog': 0.94203539999999997}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.35199496000000002}}
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2200000000000002}
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 5.2400000000000002}
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.52}
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.98}
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.6999999999999993}
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.30567935000000002, u'Compaction of SinaMiniBlog.Forwarding': 0.58998139999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.68994789999999995}}
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11983536}}
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3600000000000003}
   2012-11-21 06:35:49+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:49+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.98227600000000004, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.55468804000000005, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0640282999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.88925929999999997, u'Compaction of SinaMiniBlog.Blog': 0.91141159999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}}
   2012-11-21 06:35:49+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:49+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.2000000000000002}
   2012-11-21 06:35:50+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:50+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'data-held': 3300864062417.0, u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28115043000000001, u'Repair of SinaMiniBlog.Blog': 0.31852957999999998}, u'streams': {u'10.2.255.6': 0.35479808000000002}}
   2012-11-21 06:35:50+0800 [Weibook] DEBUG: Sending basic agent config to /1668265263/conf
   2012-11-21 06:35:50+0800 [] DEBUG: EC2 Instance type: None
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2000000000000002}
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 5.0599999999999996}
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.48}
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.899999999999999}
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.4900000000000002}
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.4100000000000001}
   2012-11-21 06:35:54+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:54+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.1799999999999997}
   2012-11-21 06:35:55+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:55+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747671913690.0, u'streams': {u'10.2.255.4': 0.74478339999999998, u'10.2.255.7': 0.064338649999999997}}
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80060070000000005, u'Compaction of SinaMiniBlog.BlogLog': 0.94531304000000005}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.35661189999999998}}
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1800000000000002}
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.9699999999999998}
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4399999999999999}
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.989999999999998}
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.5299999999999994}
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.33797700000000003, u'Compaction of SinaMiniBlog.Forwarding': 0.62380886000000002, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.69600609999999996}}
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'data-held': 2266038071590.0, u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.119886175}}
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2999999999999998}
   2012-11-21 06:35:59+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:35:59+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.56944567000000001, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0745450000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.89292693000000001, u'Compaction of SinaMiniBlog.Blog': 1.3314009, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}}
   2012-11-21 06:35:59+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:35:59+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.0899999999999999}
   2012-11-21 06:35:59+0800 [Weibook] DEBUG: Getting nodelist over thrift
   2012-11-21 06:36:00+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:00+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28122288000000001, u'Repair of SinaMiniBlog.Blog': 0.31855559999999999}, u'streams': {u'10.2.255.6': 0.35917884}}
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1699999999999999}
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.8200000000000003}
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4099999999999999}
   2012-11-21 06:36:02+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:02+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.4000000000000004}
   2012-11-21 06:36:02+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:02+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996}
   2012-11-21 06:36:04+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:04+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.0}
   2012-11-21 06:36:05+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:05+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.75113266999999995, u'10.2.255.7': 0.065497559999999996}}
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80343319999999996, u'Compaction of SinaMiniBlog.BlogLog': 0.94878240000000003}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36005956}}
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1499999999999999}
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.75}
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.45}
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.361543, u'Compaction of SinaMiniBlog.Forwarding': 0.64397895000000005, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.69999677000000005}}
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.3699999999999992}
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11993856999999999}}
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.25}
   2012-11-21 06:36:09+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:09+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.58799729999999995, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.89710294999999995, u'Compaction of SinaMiniBlog.Blog': 1.3485092000000001, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}}
   2012-11-21 06:36:09+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:09+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.8399999999999999}
   2012-11-21 06:36:09+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:10+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:10+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28128727999999997, u'Repair of SinaMiniBlog.Blog': 0.31860973999999997}, u'streams': {u'10.2.255.6': 0.36162367000000001}}
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1400000000000001}
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6900000000000004}
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4199999999999999}
   2012-11-21 06:36:11+0800 [] DEBUG: EC2 Instance ami: None
   2012-11-21 06:36:11+0800 [] INFO: Package Manager: aptitude
   2012-11-21 06:36:12+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:12+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.1799999999999997}
   2012-11-21 06:36:12+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:12+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2300000000000004}
   2012-11-21 06:36:13+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:14+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:14+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.77}
   2012-11-21 06:36:15+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:15+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.75740313999999997, u'10.2.255.7': 0.066674789999999998}}
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80602649999999998, u'Compaction of SinaMiniBlog.BlogLog': 0.9520999}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36286732999999999}}
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.21}
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6399999999999997}
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.3799999999999999}
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.149999999999999}
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'data-held': 1172553990919.0, u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.22464439999999999, u'Compaction of SinaMiniBlog.FollowLink': 0.40153450000000002, u'Compaction of SinaMiniBlog.Forwarding': 0.68902399999999997, u'Compaction of SinaMiniBlog.UserNameUid': 0.073295390000000002, u'Compaction of SinaMiniBlog.BlogLog': 0.70607715999999998}}
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.9299999999999997}
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11996788999999999}}
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.29}
   2012-11-21 06:36:18+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:18+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'data-held': 1535942604955.0, u'tasks': {u'Repair of SinaMiniBlog.UserNameUid': 0.33967935999999999}}
   2012-11-21 06:36:19+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:19+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60502279999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.90145489999999995, u'Compaction of SinaMiniBlog.Blog': 1.3666309999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}}
   2012-11-21 06:36:19+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:19+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.71}
   2012-11-21 06:36:20+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:20+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28136080000000002, u'Repair of SinaMiniBlog.Blog': 0.31864163000000001}, u'streams': {u'10.2.255.6': 0.36505553000000002}}
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1899999999999999}
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.5800000000000001}
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.51}
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.379999999999999}
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.8499999999999996}
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996}
   2012-11-21 06:36:24+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:24+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.73}
   2012-11-21 06:36:25+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:25+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747699054671.0, u'streams': {u'10.2.255.4': 0.76411640000000003, u'10.2.255.7': 0.067910760000000001}}
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80890660000000003, u'Compaction of SinaMiniBlog.BlogLog': 1.0124626000000001}}
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.3399999999999999}
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.54}
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.55}
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.43}
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.61203843000000002, u'Compaction of SinaMiniBlog.FollowLink': 0.45432818000000003, u'Compaction of SinaMiniBlog.Forwarding': 0.73092069999999998, u'Compaction of SinaMiniBlog.UserNameUid': 0.073295390000000002, u'Compaction of SinaMiniBlog.BlogLog': 0.71151423000000003}}
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.8599999999999994}
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.120037846}}
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.1699999999999999}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60923742999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.90597576000000002, u'Compaction of SinaMiniBlog.Blog': 1.3855915000000001, u'Compaction of SinaMiniBlog.UserNameUid': 0.22007544000000001}}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6800000000000002}
   2012-11-21 06:36:29+0800 [] DEBUG: Average opscenterd CPU usage: 1.75%, memory usage: 35 MB
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Getting nodelist over thrift
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.6
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.6, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.0
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.0, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.3
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.3, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.7
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.7, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.4
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.6:61621/cluster/datacenter?node_ip=10.2.255.4, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.2
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.2, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.1
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.0:61621/cluster/datacenter?node_ip=10.2.255.1, body: {}
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.5
   2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.5, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.3:61621/cluster/rack?node_ip=10.2.255.6, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.4, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.5:61621/cluster/rack?node_ip=10.2.255.0, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.7, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.2:61621/cluster/rack?node_ip=10.2.255.5, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.1, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.3, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.2, body: {}
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:30+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28142752999999998, u'Repair of SinaMiniBlog.Blog': 0.3186908}, u'streams': {u'10.2.255.6': 0.36543900000000001}}
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.3100000000000001}
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.4900000000000002}
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.51}
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.559999999999999}
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.7899999999999991}
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.0700000000000003}
   2012-11-21 06:36:34+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:34+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6200000000000001}
   2012-11-21 06:36:35+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:35+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.77013120000000002, u'10.2.255.7': 0.06900589}}
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.81165080000000001, u'Compaction of SinaMiniBlog.BlogLog': 1.0154805}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36556854999999999}}
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.29}
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3700000000000001}
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.6299999999999999}
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.59}
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.81337809999999999, u'Compaction of SinaMiniBlog.FollowLink': 0.49192249999999998, u'Compaction of SinaMiniBlog.Forwarding': 0.74063789999999996, u'Compaction of SinaMiniBlog.UserNameUid': 0.089106350000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.71590953999999996}}
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.7300000000000004}
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.12009838}}
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 3.9900000000000002}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.5:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.4:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.7:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.6:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.0:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.3:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: http://10.2.255.2:61621/os-metric/disk-space?, body: {}
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.6 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.6
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.5 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.5
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.0 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.0
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.7 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.7
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.2 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.2
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.4 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.4
   2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.3 is version u'2.10'
   2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.3
   2012-11-21 06:36:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:39+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'data-held': 1191072523699.0, u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60923742999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.91020805000000005, u'Compaction of SinaMiniBlog.Blog': 1.4013019, u'Compaction of SinaMiniBlog.UserNameUid': 0.22007544000000001}}
   2012-11-21 06:36:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:39+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6499999999999999}
   2012-11-21 06:36:40+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:40+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'data-held': 3300864072788.0, u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28150140000000001, u'Repair of SinaMiniBlog.Blog': 0.31874025}, u'streams': {u'10.2.255.6': 0.36827130000000002}}
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2599999999999998}
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.2599999999999998}
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5800000000000001}
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.219999999999999}
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.75}
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5
   2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 3.9100000000000001}
   2012-11-21 06:36:44+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4
   2012-11-21 06:36:44+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.7599999999999998}
   2012-11-21 06:36:45+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0
   2012-11-21 06:36:45+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.77678674000000003, u'10.2.25

   Posted 6 months ago #
9. 

   nickmbailey
   Key Master

   The problem just appears to be that cassandra is not running on 10.2.255.1. Can you verify this is correct?

   Posted 6 months ago #
10. 

    GodenYao
    Member

    no it was running. now .1 is back online(show data in the graph) but .4 is not.
    I did nothing. Maybe the problem is transient?
    OpsCenter Dashboard still shows "all agents connected".

    I checked .4 agent log - seems to be the similar issue as .1

    Posted 6 months ago #
11. 

    nickmbailey
    Key Master

    The log line that made me think your node was down was this:

    u'down-nodes': [u'10.2.255.1']

    That is indicating that the other nodes in your cluster are reporting .1 as being down. This can be true even if Cassandra is running. You would likely see that node as down when running 'nodetool ring' on some of the other nodes in your cluster as well.

    It seems like your cassandra cluster is having problems, perhaps related to being overloaded, where nodes are being marked as down even though the Cassandra process is still running.

    Posted 6 months ago #
12. 

    GodenYao
    Member

    we observed a heavy load of compaction jobs - that might be the cause.
    Thanks for the help.

    I had another question posted in Cassandra Forum to ask about this.
    I think we can close this thread for now.

    Thanks again!

    Posted 6 months ago #

RSS feed for this topic

Reply

You must log in to post.