DataStax Support Forums » User Favorites: godenyao

DataStax Support Forums » User Favorites: godenyao godenyao Software, Support, and Training for Apache Cassandra en-US Sun, 19 May 2013 22:53:25 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php tupton on "OpsCenter cannot detect the newly added node" http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7969 Thu, 13 Dec 2012 20:22:19 +0000 tupton 7969@http://www.datastax.com/support-forums/ jurdefur, It's possible that bootstrapping that node somehow failed. If this is the case, you can try again to get it to join. You can stop that node, wipe everything, assign a different token (just subtract one from its existing token to make it easy), and start it up again. jurdefur on "OpsCenter cannot detect the newly added node" http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7966 Thu, 13 Dec 2012 11:24:46 +0000 jurdefur 7966@http://www.datastax.com/support-forums/ Got exact the same problem here, but we only have 6gb per node and it still does not recognize the new node. It stays on "joinging" and with gossipinfo it stays on "booting". Any idee how long a join can take? GodenYao on "Cassandra is doing same validation in multiple tasks" http://www.datastax.com/support-forums/topic/cassandra-is-doing-same-validation-in-multiple-tasks#post-7721 Fri, 30 Nov 2012 03:43:37 +0000 GodenYao 7721@http://www.datastax.com/support-forums/ OK thanks for the reply. bcoverston on "Cassandra is doing same validation in multiple tasks" http://www.datastax.com/support-forums/topic/cassandra-is-doing-same-validation-in-multiple-tasks#post-7685 Tue, 27 Nov 2012 20:42:31 +0000 bcoverston 7685@http://www.datastax.com/support-forums/ This type of compaction activity is normally seen during a user triggered repair. When a this happens you will normally see multiple concurrent jobs, one for each SSTable on disk up to the number of cores that are available. It's not the same data, it is actually different files in the same column family. GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7617 Wed, 21 Nov 2012 16:54:29 +0000 GodenYao 7617@http://www.datastax.com/support-forums/ we observed a heavy load of compaction jobs - that might be the cause. Thanks for the help. I had another question posted in Cassandra Forum to ask about this. I think we can close this thread for now. Thanks again! nickmbailey on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7612 Wed, 21 Nov 2012 14:59:05 +0000 nickmbailey 7612@http://www.datastax.com/support-forums/ The log line that made me think your node was down was this: u'down-nodes': [u'10.2.255.1'] That is indicating that the other nodes in your cluster are reporting .1 as being down. This can be true even if Cassandra is running. You would likely see that node as down when running 'nodetool ring' on some of the other nodes in your cluster as well. It seems like your cassandra cluster is having problems, perhaps related to being overloaded, where nodes are being marked as down even though the Cassandra process is still running. GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7607 Wed, 21 Nov 2012 07:15:13 +0000 GodenYao 7607@http://www.datastax.com/support-forums/ no it was running. now .1 is back online(show data in the graph) but .4 is not. I did nothing. Maybe the problem is transient? OpsCenter Dashboard still shows "all agents connected". I checked .4 agent log - seems to be the similar issue as .1 nickmbailey on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7599 Tue, 20 Nov 2012 22:59:39 +0000 nickmbailey 7599@http://www.datastax.com/support-forums/ The problem just appears to be that cassandra is not running on 10.2.255.1. Can you verify this is correct? GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7598 Tue, 20 Nov 2012 22:44:22 +0000 GodenYao 7598@http://www.datastax.com/support-forums/ i restarted my opscenter service (on node .6) and the problematic one (on node .1). now dashboard shows .1 is disconnected from agent perspective. opsCenter agent log on .1 is pretty much similar to what I posted. opsCenter log on .6 is below (enabled DEBUG) ==== Log ==== wonder if there's a way to attach a file instead of pasting everything here. 2012-11-21 06:35:29+0800 [Weibook] INFO: Starting services for cluster Weibook 2012-11-21 06:35:29+0800 [] DEBUG: Getting nodelist over thrift 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.7:9160 @0x3035200> 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.6:9160 @0x3035440> 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.5:9160 @0x3035248> 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.4:9160 @0x30352d8> 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Adding connection to <CassandraNode 10.2.255.3:9160 @0x318c518> 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Waiting for describe_version() results 2012-11-21 06:35:29+0800 [Weibook] INFO: Starting CassandraCluster service 2012-11-21 06:35:29+0800 [Weibook] INFO: agent_config items: {'rollups60_ttl': 604800, 'thrift_port': 9160, 'rollups7200_ttl': 31536000, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'} 2012-11-21 06:35:29+0800 [Weibook] INFO: Starting PushService 2012-11-21 06:35:29+0800 [] INFO: OS Version: Linux version 2.6.32-5-amd64 (Debian 2.6.32-46) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Sun Sep 23 10:07:46 UTC 2012 2012-11-21 06:35:29+0800 [] INFO: CPU Info: ['2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250', '2294.250'] 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.6:9160 @0x3035440> to the pool 2012-11-21 06:35:29+0800 [] INFO: Mem Info: 96814MB 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.5:9160 @0x3035248> to the pool 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Getting nodelist over thrift 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.3:9160 @0x318c518> to the pool 2012-11-21 06:35:29+0800 [Weibook] INFO: Partitioner: org.apache.cassandra.dht.RandomPartitioner 2012-11-21 06:35:29+0800 [Weibook] INFO: Cluster Name: Weibook 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.6 ('106338239662793269832304564822427566080') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.0 ('0') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.3 ('127605887595351923798765477786913079296') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.7 ('148873535527910577765226390751398592512') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.4 ('21267647932558653966460912964485513216') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.2 ('85070591730234615865843651857942052864') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.1 ('42535295865117307932921825928971026432') 2012-11-21 06:35:29+0800 [Weibook] INFO: Recognizing new node 10.2.255.5 ('63802943797675961899382738893456539648') 2012-11-21 06:35:29+0800 [Weibook] DEBUG: Getting nodelist over thrift 2012-11-21 06:35:30+0800 [Weibook] INFO: Keyspaces: [CassandraKeyspace(name=Tortoise, column_families=[], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=TestKeyspace, column_families=['Test1'], attributes={'strategy_options': {'datacenter1': '1', 'replication_factor': None}, 'replica_placement_strategy': 'org.apache.cassandra.locator.NetworkTopologyStrategy'}), CassandraKeyspace(name=system, column_families=['schema_columnfamilies', 'NodeIdInfo', 'schema_columns', 'Versions', 'IndexInfo', 'Schema', 'schema_keyspaces', 'Migrations', 'LocationInfo', 'HintsColumnFamily'], attributes={'strategy_options': {'replication_factor': None}, 'replica_placement_strategy': 'org.apache.cassandra.locator.LocalStrategy'}), CassandraKeyspace(name=SinaMiniBlog, column_families=['User', 'BlogLog', 'MidUid', 'User2', 'Alert5', 'EditoridUid', 'Alert6', 'BlogForwarded', 'FollowerLog', 'Mark', 'MentioningInBlog', 'Replying', 'LinkInComment', 'UserFollowRank', 'UserTagByEditor', 'Blog', 'Replied', 'CommentLog', 'Forwarding', 'UserUrlUid', 'Comment', 'CidMid', 'Commented', 'UserLog', 'CrawlerData', 'Follower', 'MentioningInComment', 'MentionedInBlog', 'FollowLink', 'FollowingLog', 'CrawlerDataInfo', 'GlobalConfiguration', 'Commenting', 'LinkInBlog', 'Forwarded', 'UserNameUid', 'MentionedInComment'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=TencentMiniBlog, column_families=[], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=Weibook, column_families=['UserNameToUid', 'Exception', 'Machine', 'AccountEvent2', 'ParentTagTag', 'TortoiseDomainUrl', 'Report', 'Account', 'ExceptionLog', 'BotAccount', 'TortoiseResponse', 'CrawlerAssemblySignature2', 'RealNameInfo', 'AccountEvent', 'KeywordBag', 'CrawlerAssembly', 'TortoiseTarget2', 'Tag', 'TortoiseTarget', 'TagTypeTag'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=OpsCenter, column_families=['pdps', 'rollups86400', 'rollups7200', 'events', 'rollups300', 'events_timeline', 'settings', 'rollups60'], attributes={'strategy_options': {'replication_factor': '2'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'}), CassandraKeyspace(name=Viizii, column_families=['User', 'UserReadHistory2', 'InboxByViz', 'Viz', 'DevicesMap', 'Inbox', 'UserFavorite'], attributes={'strategy_options': {'replication_factor': '3'}, 'replica_placement_strategy': 'org.apache.cassandra.locator.SimpleStrategy'})] 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Current number of column families: 83 2012-11-21 06:35:30+0800 [] /usr/share/opscenter/lib/py/orbited/cometsession.py:467: exceptions.DeprecationWarning: twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource. 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.7:9160 @0x3035200> to the pool 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Getting nodelist over thrift 2012-11-21 06:35:30+0800 [Weibook] INFO: Initializing event storage. 2012-11-21 06:35:30+0800 [Weibook] INFO: Attempting to load all persisted alert rules 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Attempting to load all persisted scheduled job descriptions 2012-11-21 06:35:30+0800 [Weibook] INFO: Done loading persisted scheduled job descriptions 2012-11-21 06:35:30+0800 [Weibook] INFO: Done loading persisted alert rules 2012-11-21 06:35:30+0800 [Weibook] INFO: Done initializing event storage. 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Successfully added connection to <CassandraNode 10.2.255.4:9160 @0x30352d8> to the pool 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1668265263/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1758260838/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'agent_rpc_broadcast_address': u'10.2.255.0', u'tasks': {u'Repair of SinaMiniBlog.Follower': 0.39552159999999997}, u'other-devices': [u'sde', u'sdb', u'sdf', u'sdd', u'sdc'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'mars', u'network-interfaces': [u'lo', u'eth0', u'eth1', u'eth2'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.4': 0.72576505000000002, u'10.2.255.7': 0.060971785000000001}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3747670659989.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 1.6399999999999999, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.0 as the RPC address for node 10.2.255.0 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node mars is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Version: {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.0 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /987724241/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'agent_rpc_broadcast_address': u'10.2.255.7', u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.24895886, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.52155289999999999, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0430843999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.88131904999999999, u'Compaction of SinaMiniBlog.Blog': 0.86973643, u'Compaction of SinaMiniBlog.UserNameUid': 0.19911227000000001}, u'other-devices': [u'sdd', u'sdb', u'sdc', u'sde'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'uranus', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1191072487259.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 17.57, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.7 as the RPC address for node 10.2.255.7 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node uranus is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.7 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /37324024/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'agent_rpc_broadcast_address': u'10.2.255.2', u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28101482999999999, u'Repair of SinaMiniBlog.Blog': 0.31844539999999999}, u'other-devices': [u'sdd', u'sdf', u'sde', u'sdc', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'venus', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.1', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.6': 0.34552179999999999}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3300647361169.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.2800000000000002, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.2 as the RPC address for node 10.2.255.2 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node venus is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.2 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Node reported as being down: 10.2.255.1 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /214866319/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'agent_rpc_broadcast_address': u'10.2.255.6', u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.18080822999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.52787859999999998, u'Compaction of SinaMiniBlog.UserNameUid': 0.039741400000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.6784519}, u'other-devices': [u'sde', u'sdb', u'sdc', u'sdd'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'saturn', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1172552961223.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 10.74, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.6 as the RPC address for node 10.2.255.6 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node saturn is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.6 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1501295681/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'agent_rpc_broadcast_address': u'10.2.255.3', u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.7930199, u'Compaction of SinaMiniBlog.BlogLog': 0.93576820000000005}, u'other-devices': [u'sde', u'sdc', u'sdf', u'sdd', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'mercury', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.1', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {u'10.2.255.7': 0.19626610999999999, u'10.2.255.6': 0.34316286000000001}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96860, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 3228950748481.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 2.2000000000000002, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.3 as the RPC address for node 10.2.255.3 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node mercury is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.3 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /774784601/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'agent_rpc_broadcast_address': u'10.2.255.4', u'tasks': {u'Repair of SinaMiniBlog.UserNameUid': 0.33712560000000003}, u'other-devices': [u'sdb', u'sdc', u'sdd', u'sde'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'jupiter', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 1535942583196.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.75, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.4 as the RPC address for node 10.2.255.4 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node jupiter is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.4 changed its mode to normal 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Sending basic agent config to /1739664394/conf 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'agent_rpc_broadcast_address': u'10.2.255.5', u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11971977}, u'other-devices': [u'sdd', u'sde', u'sdc', u'sdb'], u'sub_os': u'.', u'commitlog-partition': u'/dev/sda2', u'data-devices': [u'md0'], u'hostname': u'earth', u'network-interfaces': [u'lo', u'eth0', u'eth1'], u'os_version': u'0.6', u'commitlog-device': u'sda', u'version': {u'search': None, u'tasktracker': None, u'dse': None, u'jobtracker': None, u'cassandra': u'1.1.6'}, u'data-partitions': [u'/dev/md0'], u'streams': {}, u'saved-caches-partition': u'/dev/sda2', u'memory': 96814, u'saved-caches-device': u'sda', u'other-partitions': [], u'data-held': 2266018552730.0, u'arch': u'amd64', u'procs': 12, u'datacenter': u'datacenter1', u'has-jna': True, u'last-seen': 0, u'os-load': 4.3899999999999997, u'mode': u'normal', u'down-nodes': [u'10.2.255.1'], u'os': u'linux', u'rack': u'rack1'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Using 10.2.255.5 as the RPC address for node 10.2.255.5 2012-11-21 06:35:30+0800 [Weibook] DEBUG: Version for node earth is {'cassandra': '1.1.6'} 2012-11-21 06:35:30+0800 [Weibook] INFO: Node 10.2.255.5 changed its mode to normal 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3399999999999999} 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1800000000000002} 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:31+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5900000000000001} 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.449999999999999} 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 10.359999999999999} 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:32+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996} 2012-11-21 06:35:33+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:33+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.4500000000000002} 2012-11-21 06:35:35+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:35+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747670669100.0, u'streams': {u'10.2.255.4': 0.73279110000000003, u'10.2.255.7': 0.062090915000000003}} 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.79573579999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.93895154999999997}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.34717967999999999}} 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3899999999999997} 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1699999999999999} 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:36+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.6200000000000001} 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.170000000000002} 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 10.09} 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.2477385, u'Compaction of SinaMiniBlog.Forwarding': 0.55636202999999995, u'Compaction of SinaMiniBlog.UserNameUid': 0.039741400000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.68375010000000003}} 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'data-held': 2266018575953.0, u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11977488}} 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3300000000000001} 2012-11-21 06:35:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:39+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.62656290000000003, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.53907627000000002, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.054003, u'Compaction of SinaMiniBlog.BlogLog': 0.88546765000000005, u'Compaction of SinaMiniBlog.Blog': 0.88998169999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.19911227000000001}} 2012-11-21 06:35:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:39+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.3300000000000001} 2012-11-21 06:35:40+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:40+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28108576000000002, u'Repair of SinaMiniBlog.Blog': 0.31848925}, u'streams': {u'10.2.255.6': 0.34963095}} 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2400000000000002} 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6799999999999997} 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:41+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5700000000000001} 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.16} 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.7599999999999998} 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:42+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3099999999999996} 2012-11-21 06:35:44+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:44+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.3099999999999996} 2012-11-21 06:35:45+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:45+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.73832750000000003, u'10.2.255.7': 0.063278249999999994}} 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.79805479999999995, u'Compaction of SinaMiniBlog.BlogLog': 0.94203539999999997}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.35199496000000002}} 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2200000000000002} 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 5.2400000000000002} 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:46+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.52} 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.98} 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.6999999999999993} 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.30567935000000002, u'Compaction of SinaMiniBlog.Forwarding': 0.58998139999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.68994789999999995}} 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11983536}} 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:47+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.3600000000000003} 2012-11-21 06:35:49+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:49+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.98227600000000004, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.55468804000000005, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0640282999999999, u'Compaction of SinaMiniBlog.BlogLog': 0.88925929999999997, u'Compaction of SinaMiniBlog.Blog': 0.91141159999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}} 2012-11-21 06:35:49+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:49+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.2000000000000002} 2012-11-21 06:35:50+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:50+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'data-held': 3300864062417.0, u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28115043000000001, u'Repair of SinaMiniBlog.Blog': 0.31852957999999998}, u'streams': {u'10.2.255.6': 0.35479808000000002}} 2012-11-21 06:35:50+0800 [Weibook] DEBUG: Sending basic agent config to /1668265263/conf 2012-11-21 06:35:50+0800 [] DEBUG: EC2 Instance type: None 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2000000000000002} 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 5.0599999999999996} 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:51+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.48} 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.899999999999999} 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.4900000000000002} 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:52+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.4100000000000001} 2012-11-21 06:35:54+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:54+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.1799999999999997} 2012-11-21 06:35:55+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:55+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747671913690.0, u'streams': {u'10.2.255.4': 0.74478339999999998, u'10.2.255.7': 0.064338649999999997}} 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80060070000000005, u'Compaction of SinaMiniBlog.BlogLog': 0.94531304000000005}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.35661189999999998}} 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1800000000000002} 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.9699999999999998} 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:35:56+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4399999999999999} 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 16.989999999999998} 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.5299999999999994} 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.33797700000000003, u'Compaction of SinaMiniBlog.Forwarding': 0.62380886000000002, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.69600609999999996}} 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'data-held': 2266038071590.0, u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.119886175}} 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:35:57+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2999999999999998} 2012-11-21 06:35:59+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:35:59+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.61570599999999998, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.56944567000000001, u'Compaction of SinaMiniBlog.FollowLink': 8.2703446000000004e-05, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0745450000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.89292693000000001, u'Compaction of SinaMiniBlog.Blog': 1.3314009, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}} 2012-11-21 06:35:59+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:35:59+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.0899999999999999} 2012-11-21 06:35:59+0800 [Weibook] DEBUG: Getting nodelist over thrift 2012-11-21 06:36:00+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:00+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28122288000000001, u'Repair of SinaMiniBlog.Blog': 0.31855559999999999}, u'streams': {u'10.2.255.6': 0.35917884}} 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1699999999999999} 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.8200000000000003} 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:01+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4099999999999999} 2012-11-21 06:36:02+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:02+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.4000000000000004} 2012-11-21 06:36:02+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:02+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996} 2012-11-21 06:36:04+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:04+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 4.0} 2012-11-21 06:36:05+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:05+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.75113266999999995, u'10.2.255.7': 0.065497559999999996}} 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80343319999999996, u'Compaction of SinaMiniBlog.BlogLog': 0.94878240000000003}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36005956}} 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1499999999999999} 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.75} 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:06+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.45} 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowLink': 0.361543, u'Compaction of SinaMiniBlog.Forwarding': 0.64397895000000005, u'Compaction of SinaMiniBlog.UserNameUid': 0.058301480000000003, u'Compaction of SinaMiniBlog.BlogLog': 0.69999677000000005}} 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.3699999999999992} 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11993856999999999}} 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:07+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.25} 2012-11-21 06:36:09+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:09+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.58799729999999995, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.89710294999999995, u'Compaction of SinaMiniBlog.Blog': 1.3485092000000001, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}} 2012-11-21 06:36:09+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:09+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.8399999999999999} 2012-11-21 06:36:09+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:10+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:10+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28128727999999997, u'Repair of SinaMiniBlog.Blog': 0.31860973999999997}, u'streams': {u'10.2.255.6': 0.36162367000000001}} 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1400000000000001} 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6900000000000004} 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:11+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.4199999999999999} 2012-11-21 06:36:11+0800 [] DEBUG: EC2 Instance ami: None 2012-11-21 06:36:11+0800 [] INFO: Package Manager: aptitude 2012-11-21 06:36:12+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:12+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 9.1799999999999997} 2012-11-21 06:36:12+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:12+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2300000000000004} 2012-11-21 06:36:13+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:14+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:14+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.77} 2012-11-21 06:36:15+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:15+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.75740313999999997, u'10.2.255.7': 0.066674789999999998}} 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80602649999999998, u'Compaction of SinaMiniBlog.BlogLog': 0.9520999}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36286732999999999}} 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.21} 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.6399999999999997} 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:16+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.3799999999999999} 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.149999999999999} 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'data-held': 1172553990919.0, u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.22464439999999999, u'Compaction of SinaMiniBlog.FollowLink': 0.40153450000000002, u'Compaction of SinaMiniBlog.Forwarding': 0.68902399999999997, u'Compaction of SinaMiniBlog.UserNameUid': 0.073295390000000002, u'Compaction of SinaMiniBlog.BlogLog': 0.70607715999999998}} 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.9299999999999997} 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.11996788999999999}} 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:17+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.29} 2012-11-21 06:36:18+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:18+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'data-held': 1535942604955.0, u'tasks': {u'Repair of SinaMiniBlog.UserNameUid': 0.33967935999999999}} 2012-11-21 06:36:19+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:19+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60502279999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.90145489999999995, u'Compaction of SinaMiniBlog.Blog': 1.3666309999999999, u'Compaction of SinaMiniBlog.UserNameUid': 0.20892951000000001}} 2012-11-21 06:36:19+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:19+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.71} 2012-11-21 06:36:20+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:20+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28136080000000002, u'Repair of SinaMiniBlog.Blog': 0.31864163000000001}, u'streams': {u'10.2.255.6': 0.36505553000000002}} 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.1899999999999999} 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.5800000000000001} 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:21+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.51} 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.379999999999999} 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.8499999999999996} 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:22+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.2699999999999996} 2012-11-21 06:36:24+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:24+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.73} 2012-11-21 06:36:25+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:25+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'data-held': 3747699054671.0, u'streams': {u'10.2.255.4': 0.76411640000000003, u'10.2.255.7': 0.067910760000000001}} 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.80890660000000003, u'Compaction of SinaMiniBlog.BlogLog': 1.0124626000000001}} 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.3399999999999999} 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.54} 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:26+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.55} 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.43} 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.61203843000000002, u'Compaction of SinaMiniBlog.FollowLink': 0.45432818000000003, u'Compaction of SinaMiniBlog.Forwarding': 0.73092069999999998, u'Compaction of SinaMiniBlog.UserNameUid': 0.073295390000000002, u'Compaction of SinaMiniBlog.BlogLog': 0.71151423000000003}} 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.8599999999999994} 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.120037846}} 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:27+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.1699999999999999} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60923742999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.90597576000000002, u'Compaction of SinaMiniBlog.Blog': 1.3855915000000001, u'Compaction of SinaMiniBlog.UserNameUid': 0.22007544000000001}} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6800000000000002} 2012-11-21 06:36:29+0800 [] DEBUG: Average opscenterd CPU usage: 1.75%, memory usage: 35 MB 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Getting nodelist over thrift 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.6 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.6" rel="nofollow">http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.6</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.0 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.0" rel="nofollow">http://10.2.255.2:61621/cluster/datacenter?node_ip=10.2.255.0</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.3 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.3" rel="nofollow">http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.3</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.7 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.7" rel="nofollow">http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.7</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.4 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.6:61621/cluster/datacenter?node_ip=10.2.255.4" rel="nofollow">http://10.2.255.6:61621/cluster/datacenter?node_ip=10.2.255.4</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.2 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.2" rel="nofollow">http://10.2.255.4:61621/cluster/datacenter?node_ip=10.2.255.2</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.1 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.0:61621/cluster/datacenter?node_ip=10.2.255.1" rel="nofollow">http://10.2.255.0:61621/cluster/datacenter?node_ip=10.2.255.1</a>, body: {} 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Collecting topology info for node 10.2.255.5 2012-11-21 06:36:29+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.5" rel="nofollow">http://10.2.255.3:61621/cluster/datacenter?node_ip=10.2.255.5</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.3:61621/cluster/rack?node_ip=10.2.255.6" rel="nofollow">http://10.2.255.3:61621/cluster/rack?node_ip=10.2.255.6</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.4" rel="nofollow">http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.4</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.5:61621/cluster/rack?node_ip=10.2.255.0" rel="nofollow">http://10.2.255.5:61621/cluster/rack?node_ip=10.2.255.0</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.7" rel="nofollow">http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.7</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.2:61621/cluster/rack?node_ip=10.2.255.5" rel="nofollow">http://10.2.255.2:61621/cluster/rack?node_ip=10.2.255.5</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.1" rel="nofollow">http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.1</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.3" rel="nofollow">http://10.2.255.0:61621/cluster/rack?node_ip=10.2.255.3</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.2" rel="nofollow">http://10.2.255.7:61621/cluster/rack?node_ip=10.2.255.2</a>, body: {} 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:30+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28142752999999998, u'Repair of SinaMiniBlog.Blog': 0.3186908}, u'streams': {u'10.2.255.6': 0.36543900000000001}} 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.3100000000000001} 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.4900000000000002} 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:31+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.51} 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.559999999999999} 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.7899999999999991} 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:32+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 4.0700000000000003} 2012-11-21 06:36:34+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:34+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6200000000000001} 2012-11-21 06:36:35+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:35+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.77013120000000002, u'10.2.255.7': 0.06900589}} 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'tasks': {u'Compaction of SinaMiniBlog.Blog': 0.81165080000000001, u'Compaction of SinaMiniBlog.BlogLog': 1.0154805}, u'streams': {u'10.2.255.7': 0.19714265, u'10.2.255.6': 0.36556854999999999}} 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.29} 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.3700000000000001} 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:36+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.6299999999999999} 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.59} 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'tasks': {u'Compaction of SinaMiniBlog.FollowerLog': 0.81337809999999999, u'Compaction of SinaMiniBlog.FollowLink': 0.49192249999999998, u'Compaction of SinaMiniBlog.Forwarding': 0.74063789999999996, u'Compaction of SinaMiniBlog.UserNameUid': 0.089106350000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.71590953999999996}} 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.7300000000000004} 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Sending basic agent config to /1152066162/conf 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'tasks': {u'Repair of SinaMiniBlog.Blog': 0.12009838}} 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:37+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 3.9900000000000002} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.5:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.5:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.4:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.4:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.7:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.7:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.6:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.6:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.0:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.0:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.3:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.3:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Performing HTTP request: <a href="http://10.2.255.2:61621/os-metric/disk-space?" rel="nofollow">http://10.2.255.2:61621/os-metric/disk-space?</a>, body: {} 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.6 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.6 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.5 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.5 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.0 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.0 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.7 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.7 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.2 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.2 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.4 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.4 2012-11-21 06:36:38+0800 [Weibook] INFO: Agent for ip 10.2.255.3 is version u'2.10' 2012-11-21 06:36:38+0800 [Weibook] DEBUG: Sending agent config to node %s 10.2.255.3 2012-11-21 06:36:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:39+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'data-held': 1191072523699.0, u'tasks': {u'Compaction of SinaMiniBlog.MidUid': 0.64754915000000002, u'Compaction of SinaMiniBlog.FollowingLog': 0.99295865999999999, u'Compaction of SinaMiniBlog.CrawlerData': 0.0027273054999999999, u'Compaction of SinaMiniBlog.Forwarding': 0.60923742999999997, u'Compaction of SinaMiniBlog.FollowLink': 0.34173563000000001, u'Compaction of SinaMiniBlog.Forwarded': 0.44267731999999999, u'Compaction of SinaMiniBlog.Comment': 1.0826747000000001, u'Compaction of SinaMiniBlog.BlogLog': 0.91020805000000005, u'Compaction of SinaMiniBlog.Blog': 1.4013019, u'Compaction of SinaMiniBlog.UserNameUid': 0.22007544000000001}} 2012-11-21 06:36:39+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:39+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.6499999999999999} 2012-11-21 06:36:40+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:40+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'data-held': 3300864072788.0, u'tasks': {u'Repair of SinaMiniBlog.UserLog': 0.28150140000000001, u'Repair of SinaMiniBlog.Blog': 0.31874025}, u'streams': {u'10.2.255.6': 0.36827130000000002}} 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.3 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.3, data is {u'os-load': 2.2599999999999998} 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.2 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.2, data is {u'os-load': 4.2599999999999998} 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:41+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'os-load': 1.5800000000000001} 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.7 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.7, data is {u'os-load': 17.219999999999999} 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.6 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.6, data is {u'os-load': 8.75} 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.5 2012-11-21 06:36:42+0800 [Weibook] DEBUG: Found node for 10.2.255.5, data is {u'os-load': 3.9100000000000001} 2012-11-21 06:36:44+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.4 2012-11-21 06:36:44+0800 [Weibook] DEBUG: Found node for 10.2.255.4, data is {u'os-load': 3.7599999999999998} 2012-11-21 06:36:45+0800 [Weibook] DEBUG: Processing node detail from agent for 10.2.255.0 2012-11-21 06:36:45+0800 [Weibook] DEBUG: Found node for 10.2.255.0, data is {u'streams': {u'10.2.255.4': 0.77678674000000003, u'10.2.25 nickmbailey on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7597 Tue, 20 Nov 2012 22:22:30 +0000 nickmbailey 7597@http://www.datastax.com/support-forums/ That log is indicating that the agent is having a problem connecting to cassandra and determining its ip address. A couple things could cause that. Can you change the agent log level to DEBUG and attach that log, as well as attaching the log from the main opscenter machine for a few minutes after you restart the agent. GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7596 Tue, 20 Nov 2012 22:17:05 +0000 GodenYao 7596@http://www.datastax.com/support-forums/ after i restarted all agents, some are back to inserting data. some are not. here's the problematic one's log: It seems that it has some issues and keep waiting... Any extra log you need for diagnosing? Starting OpsCenter agent monitor opscenter_agent_monitor. INFO [main] 2012-11-21 05:54:59,259 Loading conf files: /var/lib/opscenter-agent/conf/address.yaml INFO [main] 2012-11-21 05:54:59,336 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_05 INFO [main] 2012-11-21 05:54:59,336 Waiting for the config from OpsCenter INFO [main] 2012-11-21 05:54:59,337 SSL communication is disabled INFO [main] 2012-11-21 05:54:59,337 Creating stomp connection to 10.2.255.6:61620 INFO [StompConnection receiver] 2012-11-21 05:54:59,340 Reconnecting in 0s. INFO [main] 2012-11-21 05:54:59,341 Attempting to determine Cassandra's broadcast address through JMX INFO [StompConnection receiver] 2012-11-21 05:54:59,345 Connected to 10.2.255.6:61620 INFO [Initialization] 2012-11-21 05:54:59,348 Sleeping for 2s before trying to determine IP over JMX again INFO [StompConnection receiver] 2012-11-21 05:54:59,414 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"} INFO [StompConnection receiver] 2012-11-21 05:54:59,420 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"} INFO [Initialization] 2012-11-21 05:55:01,350 New JMX connection (127.0.0.1:7199) INFO [Initialization] 2012-11-21 05:55:01,504 Sleeping for 4s before trying to determine IP over JMX again INFO [StompConnection receiver] 2012-11-21 05:55:01,554 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, OpsCenter"} INFO [Initialization] 2012-11-21 05:55:05,505 New JMX connection (127.0.0.1:7199) INFO [Initialization] 2012-11-21 05:55:05,515 Sleeping for 8s before trying to determine IP over JMX again nickmbailey on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7595 Tue, 20 Nov 2012 22:14:51 +0000 nickmbailey 7595@http://www.datastax.com/support-forums/ When the agent encounters an error like that it will put the operation back in its queue of writes and then wait 5 seconds before starting to process inserts again. You won't see any special logging for the retry however. There could be another issue with your agents that is causing them to stop writing data. I would need to inspect the logs to be sure though. GodenYao on "Cassandra is doing same validation in multiple tasks" http://www.datastax.com/support-forums/topic/cassandra-is-doing-same-validation-in-multiple-tasks#post-7594 Tue, 20 Nov 2012 22:09:30 +0000 GodenYao 7594@http://www.datastax.com/support-forums/ we used to use size tiered compaction then alter column family to leveled compaction. it seems that each node has a lot of compaction jobs in our current system. (we originally had 4 nodes, recently added 4 so 8 in total now) e.g. this node has 2143 pending tasks. nodetool compactionstats pending tasks: 2143 compaction type keyspace column family bytes compacted bytes total progress Compaction XData UserNameUid 9849441339 20298356110 48.52% Validation XData UserNameUid 202139799576 758257897889 26.66% Validation XData UserNameUid 190699406201 760934574422 25.06% Validation XData UserNameUid 170953615792 758236279998 22.55% Validation XData UserNameUid 209708959238 761914537718 27.52% Active compaction remaining time : n/a however, we see the same column family being in validation in multiple task progress. The question is that why cassandra seems doing same thing again and again over the same set of data? the column family UserNameUid has 760GB data in total and it seems ridiculous to perform multiple concurrent jobs doing the same thing on the same data. GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7593 Tue, 20 Nov 2012 21:54:38 +0000 GodenYao 7593@http://www.datastax.com/support-forums/ When you said it'll retry - when will that happen - from the log i don't see it retry and that's when I noticed I don't have data from the nodes. The only thing to bring it back is to restart agent service. GodenYao on "OpsCenter Agent Exception" http://www.datastax.com/support-forums/topic/opscenter-agent-exception#post-7592 Tue, 20 Nov 2012 21:52:50 +0000 GodenYao 7592@http://www.datastax.com/support-forums/ it's kinda frequent - maybe it's related to our recent schema change. I'll keep watching.