Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Nodetool Effective-Ownership > 100%

(4 posts) (2 voices)

Started 9 months ago by AndyBallingall
Latest reply from AndyBallingall
This topic is not a support question

Tags:

AndyBallingall
Member

I've been experimenting with a 4 node cluster running on my Windows PC.

Yesterday, I'm pretty sure 'nodetool ring' reported that the ownership was split equally among the 4 nodes, 25% each.

Last night, my machine rebooted after a system update and I had to restart the 4 node VMs from a hard reset.

Once I got cassandra up and running again, the 'effective-ownership' reports that each node shares 50% - that's 200% in total!

What's going on?

I'm not sure nodetool said 'Effective-Ownership' yesterday... is this a clue that something's gone wrong?

Here's the output of nodetool ring:

[root@sweeney-delta ~]# nodetool ring
Address DC Rack Status State Load Effective-Ownership Token
127605887595351923798765477786913079296
192.9.200.157 datacenter1 rack1 Up Normal 80.63 KB 50.00% 0
192.9.200.6 datacenter1 rack1 Up Normal 68.74 KB 50.00% 42535295865117307932921825928971026432
192.9.200.115 datacenter1 rack1 Up Normal 99.43 KB 50.00% 85070591730234615865843651857942052864
192.9.200.130 datacenter1 rack1 Up Normal 55.73 KB 50.00% 127605887595351923798765477786913079296

The initialisation log from /var/log/cassandra/cassandra.log for one of the nodes is below.

(I can't see anything erroring there)

Thanks in advance,
Andy

----

INFO 09:56:37,170 Logging initialized
INFO 09:56:37,178 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_33
INFO 09:56:37,178 Heap size: 101711872/101711872
INFO 09:56:37,182 Classpath: /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-1.1.2.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-1.1.2.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-1.1.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra//lib/jamm-0.2.5.jar
INFO 09:56:37,709 JNA mlockall successful
INFO 09:56:37,718 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 09:56:37,887 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 09:56:38,192 Global memtable threshold is enabled at 32MB
INFO 09:56:38,604 Initializing key cache with capacity of 4 MBs.
INFO 09:56:38,621 Scheduling key cache save to each 14400 seconds (going to save all keys).
INFO 09:56:38,622 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
INFO 09:56:38,626 Scheduling row cache save to each 0 seconds (going to save all keys).
INFO 09:56:38,784 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-15 (1409 bytes)
INFO 09:56:38,801 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-14 (1325 bytes)
INFO 09:56:38,802 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-13 (8140 bytes)
INFO 09:56:38,814 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache
INFO 09:56:38,822 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hd-1 (1232 bytes)
INFO 09:56:38,830 Opening /var/lib/cassandra/data/system/Versions/system-Versions-hd-2 (247 bytes)
INFO 09:56:38,832 Opening /var/lib/cassandra/data/system/Versions/system-Versions-hd-1 (247 bytes)
INFO 09:56:38,833 Opening /var/lib/cassandra/data/system/Versions/system-Versions-hd-3 (247 bytes)
INFO 09:56:38,841 reading saved cache /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache
INFO 09:56:38,843 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-10 (246 bytes)
INFO 09:56:38,845 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-9 (246 bytes)
INFO 09:56:38,855 reading saved cache /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
INFO 09:56:38,857 Opening /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hd-17 (500 bytes)
INFO 09:56:39,261 Opening /var/lib/cassandra/data/OpsCenter/events/OpsCenter-events-hd-1 (179 bytes)
INFO 09:56:39,268 Opening /var/lib/cassandra/data/OpsCenter/events_timeline/OpsCenter-events_timeline-hd-1 (183 bytes)
INFO 09:56:39,273 completed pre-loading (10 keys) key cache.
INFO 09:56:39,389 Replaying /var/lib/cassandra/commitlog/CommitLog-179731308142.log, /var/lib/cassandra/commitlog/CommitLog-179887522269.log, /var/lib/cassandra/commitlog/CommitLog-179893848845.log, /var/lib/cassandra/commitlog/CommitLog-179890119025.log, /var/lib/cassandra/commitlog/CommitLog-179902583184.log, /var/lib/cassandra/commitlog/CommitLog-179892431840.log
INFO 09:56:39,394 Replaying /var/lib/cassandra/commitlog/CommitLog-179731308142.log
INFO 09:56:39,425 Finished reading /var/lib/cassandra/commitlog/CommitLog-179731308142.log
INFO 09:56:39,425 Replaying /var/lib/cassandra/commitlog/CommitLog-179887522269.log
INFO 09:56:39,426 Finished reading /var/lib/cassandra/commitlog/CommitLog-179887522269.log
INFO 09:56:39,426 Replaying /var/lib/cassandra/commitlog/CommitLog-179893848845.log
INFO 09:56:39,446 Finished reading /var/lib/cassandra/commitlog/CommitLog-179893848845.log
INFO 09:56:39,446 Replaying /var/lib/cassandra/commitlog/CommitLog-179890119025.log
INFO 09:56:39,447 Finished reading /var/lib/cassandra/commitlog/CommitLog-179890119025.log
INFO 09:56:39,447 Replaying /var/lib/cassandra/commitlog/CommitLog-179902583184.log
INFO 09:56:39,447 Finished reading /var/lib/cassandra/commitlog/CommitLog-179902583184.log
INFO 09:56:39,448 Replaying /var/lib/cassandra/commitlog/CommitLog-179892431840.log
INFO 09:56:39,448 Finished reading /var/lib/cassandra/commitlog/CommitLog-179892431840.log
INFO 09:56:39,449 Log replay complete, 0 replayed mutations
INFO 09:56:39,472 Cassandra version: 1.1.2
INFO 09:56:39,472 Thrift API version: 19.32.0
INFO 09:56:39,478 CQL supported versions: 2.0.0,3.0.0-beta1 (default: 2.0.0)
INFO 09:56:39,529 Loading persisted ring state
INFO 09:56:39,540 Starting up server gossip
INFO 09:56:39,544 Enqueuing flush of Memtable-LocationInfo@1956297961(41/51 serialized/live bytes, 1 ops)
INFO 09:56:39,544 Writing Memtable-LocationInfo@1956297961(41/51 serialized/live bytes, 1 ops)
INFO 09:56:39,729 Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hd-18-Data.db (80 bytes) for commitlog position ReplayPosition(segmentId=929465670550, position=363)
INFO 09:56:39,778 Starting Messaging Service on port 7000
INFO 09:56:39,803 Using saved token 0
INFO 09:56:39,805 Enqueuing flush of Memtable-LocationInfo@1190265908(62/77 serialized/live bytes, 2 ops)
INFO 09:56:39,805 Writing Memtable-LocationInfo@1190265908(62/77 serialized/live bytes, 2 ops)
INFO 09:56:39,865 Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hd-19-Data.db (148 bytes) for commitlog position ReplayPosition(segmentId=929465670550, position=529)
INFO 09:56:39,868 Node sweeney-delta.london.thefoundry.co.uk/192.9.200.157 state jump to normal
INFO 09:56:39,876 Bootstrap/Replace/Move completed! Now serving reads.
INFO 09:56:39,877 Will not load MX4J, mx4j-tools.jar is not in the classpath

Posted 9 months ago #
AndyBallingall
Member

In response to my own post, I decided to stop the cluster, delete all the data, and restart it.

Now, when I run 'nodetool ring', it looks ok:

Address DC Rack Status State Load Owns Token
127605887595351923798765477786913079296
192.9.200.157 datacenter1 rack1 Up Normal 11.13 KB 25.00% 0
192.9.200.6 datacenter1 rack1 Up Normal 15.55 KB 25.00% 42535295865117307932921825928971026432
192.9.200.115 datacenter1 rack1 Up Normal 15.52 KB 25.00% 85070591730234615865843651857942052864
192.9.200.130 datacenter1 rack1 Up Normal 15.55 KB 25.00% 127605887595351923798765477786913079296

The column with percentages has changed to saying 'Owns' again (like it did yesterday), and the total adds up to 100%.

What does 'Effective-Ownership' mean, and what does it mean if it adds up to 200%?

Posted 9 months ago #
thobbs
Administrator

"Effective Ownership" is referring to what percentage of the ring the node is a replica for for a given keyspace, because it depends on the replication strategy and strategy options. So, if your keyspace has a replication factor of 2 (which the OpsCenter keyspace does), each node of the four nodes is a replica for 50% of the ring.

"Owns" just covers the "primary range" for a node, which is the range counter-clockwise from that node until the next node. I'm not sure why it switched back to showing this. Perhaps the OpsCenter keyspace is gone now?

Posted 9 months ago #
AndyBallingall
Member

Hi Throbs,

I understand what happened now. Yes, I had pointed OpsCenter at the cluster, so that would explain the 50%. So everything was operating as intended all along, and it's nice to see it recovering from that hard reset.

The 'nodetool ring' output appears to show 'Owns' as the percentage heading when you haven't put any data in. It's now reading 'Effective-Ownership' again.

Thanks for your help.

Posted 9 months ago #

RSS feed for this topic