DataStax Support Forums » Topic: Opscenter new node detection issue

DataStax Support Forums » Topic: Opscenter new node detection issue http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue Software, Support, and Training for Apache Cassandra en-US Sun, 26 May 2013 00:43:18 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue/page/2#post-958 Fri, 30 Dec 2011 20:44:32 +0000 Anonymous 958@http://www.datastax.com/support-forums/ Thanks Nick! Actually when we first created this instance, we were playing around with examples available, unknowingly copy pasted the code to create a new keyspace, and that is how 'datacenter1' and DC2 got creeped in. I just dropped the keyspaces as they were dummy testing ones. Restarted the Opscenter, It works great!! Thank you guys!! nickmbailey on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue/page/2#post-957 Fri, 30 Dec 2011 20:03:02 +0000 nickmbailey 957@http://www.datastax.com/support-forums/ I see the problem from that output. There are two keyspaces that are causing the problem. The keyspace testspace0 is set to have two replicas in 'datacenter1'. Also the keyspace twissandra is set to have 3 replicas in 'DC2'. The problem is that you do not have any nodes in datacenter1 or DC2. This is a bug in OpsCenter and will be fixed in the next release, for now though this configuration will not work with OpsCenter. The workaround would be to update these keyspaces to replicate to an existing datacenter, in your case 'us-east'. Alternatively you could update the keyspaces to use SimpleStrategy which is not datacenter aware, or simply delete the keyspaces. Regarding your question about connected nodes, that number is indicating how many agents have connected to OpsCenter, The problem may also be related to the replication bug though. Once that is resolved please see if that updates correctly and let us know. Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-956 Fri, 30 Dec 2011 19:55:55 +0000 Anonymous 956@http://www.datastax.com/support-forums/ Also Nick, the Opscenter in the browser, I see none of the node are connected. It says 0 of 0 nodes are connected, whereas in ec2 aws console and in the nodetool, i see everything is connected, shared & owns equal load. Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-955 Fri, 30 Dec 2011 19:49:48 +0000 Anonymous 955@http://www.datastax.com/support-forums/ Nick, By keyspace, do you mean the Opscneter Keyspace, or all the keyspaces in general. i did a show keyspace in the cassandra cli, Here it goes. Welcome to Cassandra CLI version 1.0.5 Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. [default@unknown] show keyspaces ; Keyspace: Keyspace1: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: Counter1 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: Standard1 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.AsciiType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: Super1 (Super) Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.AsciiType/org.apache.cassandra.db.marshal.AsciiType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: SuperCounter1 (Super) Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: tvl_test2 "This CF has no comment. Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: OpsCenter: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:2] Column Families: ColumnFamily: events "OpsCenter raw event storage" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 8/12 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: events_timeline "OpsCenter event timelines" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.LongType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 5.0/0 GC grace seconds: 864000 Compaction min/max thresholds: 4/8 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: pdps Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 300.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: rollups300 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 4/16 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: rollups60 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: rollups7200 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 2/8 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: rollups86400 Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 2/8 Read repair chance: 0.25 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: settings "OpsCenter settings" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider Key cache size / save period in seconds: 50.0/43200 GC grace seconds: 864000 Compaction min/max thresholds: 8/12 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: fengspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: domain_dimension "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: publisher_dimension "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: test_qa "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: test_qa1 "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: test_qa2 "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: test_qa3 "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: users Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Column Metadata: Column Name: birth_year Validation Class: org.apache.cassandra.db.marshal.LongType Column Name: email Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: full_name Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: gender Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: state Validation Class: org.apache.cassandra.db.marshal.UTF8Type Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: qaspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: users Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 Replicate on write: true Built indexes: [] Column Metadata: Column Name: dob_year Validation Class: org.apache.cassandra.db.marshal.LongType Column Name: password Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: state Validation Class: org.apache.cassandra.db.marshal.UTF8Type Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: system: Replication Strategy: org.apache.cassandra.locator.LocalStrategy Durable Writes: false Options: [replication_factor:1] Column Families: ColumnFamily: HintsColumnFamily (Super) "hinted handoff data" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: IndexInfo "indexes that have been completed" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: LocationInfo "persistent metadata for the local node" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: Migrations "individual schema mutations" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: NodeIdInfo "nodeId and their metadata" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: Schema "current state of the schema" Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: Versions "server version information" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 0.01/0 GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: true Built indexes: [] Column Metadata: Column Name: version Validation Class: org.apache.cassandra.db.marshal.UTF8Type Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: testspace0: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [datacenter1:2] Column Families: ColumnFamily: publisher_dimension "This CF has no comment. Why are you looking here anyway? Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: test_qa "This CF has no comment. Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [test_qa.test_qa_636f6c41_idx, test_qa.test_qa_636f6c42_idx, test_qa.test_qa_636f6c43_idx] Column Metadata: Column Name: colA (636f6c41) Validation Class: org.apache.cassandra.db.marshal.UTF8Type Index Name: test_qa_636f6c41_idx Index Type: KEYS Column Name: colB (636f6c42) Validation Class: org.apache.cassandra.db.marshal.UTF8Type Index Name: test_qa_636f6c42_idx Index Type: KEYS Column Name: colC (636f6c43) Validation Class: org.apache.cassandra.db.marshal.UTF8Type Index Name: test_qa_636f6c43_idx Index Type: KEYS Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: tvl_test2 "This CF has no comment. Please add one!" Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy ColumnFamily: users Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] Column Metadata: Column Name: birth_year Validation Class: org.apache.cassandra.db.marshal.LongType Column Name: email Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: full_name Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: gender Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: state Validation Class: org.apache.cassandra.db.marshal.UTF8Type Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Keyspace: twissandra: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [DC2:3] Column Families: ColumnFamily: users Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 0.1 Replicate on write: true Built indexes: [] Column Metadata: Column Name: birth_year Validation Class: org.apache.cassandra.db.marshal.LongType Column Name: gender Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: password Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: session_token Validation Class: org.apache.cassandra.db.marshal.UTF8Type Column Name: state Validation Class: org.apache.cassandra.db.marshal.UTF8Type Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy nickmbailey on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-954 Fri, 30 Dec 2011 19:16:14 +0000 nickmbailey 954@http://www.datastax.com/support-forums/ Nish, What are your replication settings? (What is the output of nodetool cfstats) That error is known to occur when the replication settings for a keyspace indicate that replicas belong in a datacenter that currently has 0 nodes in it. If that is the case, updating the replication settings for that keyspace should fix the problem. Otherwise can you paste the output of cfstats here so we can determine if there is another problem? -Nick Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-953 Fri, 30 Dec 2011 19:10:12 +0000 Anonymous 953@http://www.datastax.com/support-forums/ Ok, here you go. 2011-12-30 19:03:52+0000 [] INFO: Received SIGTERM, shutting down. 2011-12-30 19:03:52+0000 [] INFO: Stopping CassandraCluster service 2011-12-30 19:03:52+0000 [] INFO: (TCP Port 61620 Closed) 2011-12-30 19:03:52+0000 [] INFO: (TCP Port 61619 Closed) 2011-12-30 19:03:52+0000 [] INFO: Stopping factory <morbid.morbid.StompFactory instance at 0x25110e0> 2011-12-30 19:03:52+0000 [] INFO: (TCP Port 8888 Closed) 2011-12-30 19:03:52+0000 [] INFO: Stopping factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x24e8c68> 2011-12-30 19:03:52+0000 [] INFO: Main loop terminated. 2011-12-30 19:03:52+0000 [] INFO: Server Shut Down. 2011-12-30 19:03:52+0000 [] INFO: Log opened. 2011-12-30 19:03:52+0000 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.6) starting up. 2011-12-30 19:03:52+0000 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor. 2011-12-30 19:03:52+0000 [] INFO: set uid/gid 0/0 2011-12-30 19:03:52+0000 [] INFO: Logging level set to 'debug' 2011-12-30 19:03:52+0000 [] INFO: OpsCenterdService startService 2011-12-30 19:03:52+0000 [] INFO: OpsCenter version: 1.3.1 2011-12-30 19:03:52+0000 [] INFO: Compatible agent version: 2.4 2011-12-30 19:03:52+0000 [] DEBUG: Logging user config options: 2011-12-30 19:03:52+0000 [] DEBUG: jmx : [('port', '7199')], logging : [('level', 'DEBUG')], webserver : [('interface', '0.0.0.0'), ('port', '8888')], authentication : [], cassandra_metrics : [], cassandra : [('seed_hosts', '10.68.177.38')], 2011-12-30 19:03:52+0000 [] INFO: HTTP BASIC authentication disabled 2011-12-30 19:03:52+0000 [] DEBUG: Getting nodelist over thrift 2011-12-30 19:03:52+0000 [] INFO: agent_config items: {'metrics_enabled': 1, 'http_server_port': 61621, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199} 2011-12-30 19:03:52+0000 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-12-30 19:03:52+0000 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x24e8bd8> 2011-12-30 19:03:52+0000 [] DEBUG: Adding connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> 2011-12-30 19:03:52+0000 [] DEBUG: Adding connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> 2011-12-30 19:03:52+0000 [] DEBUG: Adding connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> 2011-12-30 19:03:52+0000 [] DEBUG: Adding connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> 2011-12-30 19:03:52+0000 [] DEBUG: Adding connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> 2011-12-30 19:03:52+0000 [] DEBUG: Waiting for describe_version() results 2011-12-30 19:03:52+0000 [] INFO: Starting CassandraCluster service 2011-12-30 19:03:52+0000 [] INFO: morbid.morbid.StompFactory starting on 61619 2011-12-30 19:03:52+0000 [] INFO: Starting factory <morbid.morbid.StompFactory instance at 0x2511320> 2011-12-30 19:03:52+0000 [] INFO: morbid.morbid.StompFactory starting on 61620 2011-12-30 19:03:52+0000 [] INFO: Starting PushService 2011-12-30 19:03:52+0000 [] DEBUG: Waiting for a successful thrift call 2011-12-30 19:03:52+0000 [] INFO: CPU Info: ['2000.070', '2000.070', '2000.070', '2000.070'] 2011-12-30 19:03:52+0000 [] INFO: Mem Info: 15013MB 2011-12-30 19:03:52+0000 [] DEBUG: Successfully added connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> to the pool 2011-12-30 19:03:52+0000 [] DEBUG: Successfully added connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> to the pool 2011-12-30 19:03:52+0000 [] DEBUG: Successfully added connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> to the pool 2011-12-30 19:03:52+0000 [] DEBUG: Successfully added connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> to the pool 2011-12-30 19:03:52+0000 [] DEBUG: Successfully added connection to <CassandraNode 10.68.177.XX:9160 @0x251cfc8> to the pool 2011-12-30 19:03:52+0000 [] INFO: OS Version: Linux version 2.6.35-30-virtual (buildd@allspice) (gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5) ) #60-Ubuntu SMP Mon Sep 19 21:17:09 UTC 2011 2011-12-30 19:03:52+0000 [] DEBUG: Getting nodelist over thrift 2011-12-30 19:03:52+0000 [] INFO: Cluster Name: bidev 2011-12-30 19:03:52+0000 [] INFO: Partitioner: org.apache.cassandra.dht.RandomPartitioner 2011-12-30 19:03:52+0000 [] INFO: Package Manager: aptitude 2011-12-30 19:03:52+0000 [] DEBUG: EC2 Instance type: m1.xlarge 2011-12-30 19:03:52+0000 [] DEBUG: EC2 Instance: datastax-use/datastax_clustering_ami_2.1.manifest.xml 2011-12-30 19:03:52+0000 [] INFO: Unhandled error in Deferred: 2011-12-30 19:03:52+0000 [] Unhandled Error Traceback (most recent call last): File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 388, in errback self._startRunCallbacks(fail) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks self._runCallbacks() File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1076, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.6/dist-packages/opscenterd/Cluster.py", line 406, in startService File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.6/dist-packages/opscenterd/Cluster.py", line 573, in collectNodeList File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks result = g.send(result) File "/usr/lib/python2.6/dist-packages/opscenterd/CassandraService.py", line 179, in getTokenList File "/usr/lib/python2.6/dist-packages/opscenterd/CassandraService.py", line 191, in processTokenLists exceptions.IndexError: list index out of range 2011-12-30 19:03:53+0000 [] INFO: Done loading metric configuration. 2011-12-30 19:03:53+0000 [] INFO: Done preloading last rollup times. 2011-12-30 19:03:53+0000 [] /usr/share/opscenter/lib/py/orbited/cometsession.py:467: exceptions.DeprecationWarning: twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource. 2011-12-30 19:03:53+0000 [] INFO: Loading all persisted alert rules 2011-12-30 19:03:53+0000 [] DEBUG: Getting nodelist over thrift 2011-12-30 19:03:53+0000 [] INFO: Unhandled error in Deferred: 2011-12-30 19:03:53+0000 [] Unhandled Error Traceback (most recent call last): File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 388, in errback self._startRunCallbacks(fail) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks self._runCallbacks() File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1076, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/lib/python2.6/dist-packages/opscenterd/Cluster.py", line 573, in collectNodeList File "/usr/share/opscenter/lib/py-debian/2.6/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks result = g.send(result) File "/usr/lib/python2.6/dist-packages/opscenterd/CassandraService.py", line 179, in getTokenList File "/usr/lib/python2.6/dist-packages/opscenterd/CassandraService.py", line 191, in processTokenLists exceptions.IndexError: list index out of range mbulman on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-949 Fri, 30 Dec 2011 15:13:33 +0000 mbulman 949@http://www.datastax.com/support-forums/ I'm truly at a loss here. Would it be possible for you to change your opscenter log level to DEBUG, restart opscenter (sudo service opscenterd restart), and paste/attach the contents of the log file since that restart here? Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-944 Thu, 29 Dec 2011 22:06:18 +0000 Anonymous 944@http://www.datastax.com/support-forums/ FYI, am using DataStax Auto-Clustering AMI 2.1 (ami-fd23ec94) instances running on all the nodes. Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-943 Thu, 29 Dec 2011 19:50:41 +0000 Anonymous 943@http://www.datastax.com/support-forums/ Since the Opscenter usually runs on the first machine on the cluster. (here it is A). I rebooted the machine from the AWS console. (Rightclick ->Reboot option). And then saw the A:8888/opscenter/index.html is not detecting that change on the browser. mbulman on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-942 Thu, 29 Dec 2011 19:33:24 +0000 mbulman 942@http://www.datastax.com/support-forums/ When you originally mentioned rebooting opscenter, can you elaborate on exactly what you did? Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-941 Thu, 29 Dec 2011 19:21:18 +0000 Anonymous 941@http://www.datastax.com/support-forums/ AMI log: ~/datastax_ami/ami.log Cassandra log: /var/log/cassandra/system.log Address DC Rack Status State Load Owns Token 1488735355279105777652263907 A us-east 1b Up Normal 4.94 GB 12.50% 0 B1 us-east 1b Up Normal 5.05 GB 12.50% 2126764793255865396646091296 C us-east 1b Up Normal 5.39 GB 12.50% 4253529586511730793292182592 D us-east 1b Up Normal 5.71 GB 12.50% 6380294379767596189938273889 E us-east 1b Up Normal 5.25 GB 12.50% 8507059173023461586584365185 F us-east 1b Up Normal 4.74 GB 12.50% 1063382396627932698323045648 G us-east 1b Up Normal 4.67 GB 12.50% 1276058875953519237987654777 H us-east 1b Up Normal 5.57 GB 12.50% 1488735355279105777652263907 There are 8 nodes in total. I specifically mentioned B1 node because there was B before and it was replaced by B1. mbulman on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-940 Thu, 29 Dec 2011 18:50:30 +0000 mbulman 940@http://www.datastax.com/support-forums/ Would you mind pasting the output of nodetool ring here? Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-939 Thu, 29 Dec 2011 18:46:56 +0000 Anonymous 939@http://www.datastax.com/support-forums/ I ran nodetool ring on the old one as well as the new one. The load is equally distributed and the new one owns equally as well. Looks all good. mbulman on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-938 Thu, 29 Dec 2011 18:37:59 +0000 mbulman 938@http://www.datastax.com/support-forums/ Which node are you running nodetool ring on? If it is just the new one, can you try running on one of the others in the cluster? Anonymous on "Opscenter new node detection issue" http://www.datastax.com/support-forums/topic/opscenter-new-node-detection-issue#post-937 Thu, 29 Dec 2011 18:35:58 +0000 Anonymous 937@http://www.datastax.com/support-forums/ Yes, you are exactly right. By, 'Does not detect', I intend to say, It is not showing up in the cluster view- either in ring, physical or list view. No obvious errors in opscenter log, except for the decommissioned node that went down, and I separated it from the List view. (it is no longer connected). Also I wish to say there is bug here. Although, in the List view I could see two separated lines, (one connected and another decommissioned), In the ring view they appear to be connected. Most important of all, I am not able to see the the new node in the Opscenter. When I manually logg into the box, and wun the nodetool -ring command, I could see all them, (including the new node). Is there anything I need to change in the Opscenter setting logs? Please advise.. Thanks, Nish