Hi, I keep on getting the following message in system.log. Is there some kind of memory leak and should I be worried?
WARN [ScheduledTasks:1] 2012-10-22 14:06:12,469 GCInspector.java (line 145) Heap is 0.7878062075264272 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Cassandra 1.0.7
=== Hardware ===
96GB memory
Inteal dual socket X5660 @ 2.80GHz
Dell 710
== java -version ==
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
==cassandra.yaml==
cluster_name: 'Production'
initial_token: 0
hinted_handoff_enabled: true
max_hint_window_in_ms: 3600000 # one hour
hinted_handoff_throttle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:
- /data/cass1
commitlog_directory: /trans_logs/cass1
saved_caches_directory: /trans_logs/cass1/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "10.0.0.3,10.0.0.4,10.0.0.5"
flush_largest_memtables_at: 0.60
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: 0.6
concurrent_reads: 32
concurrent_writes: 32
memtable_flush_queue_size: 4
sliced_buffer_size_in_kb: 64
storage_port: 7000
ssl_storage_port: 7001
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: true
snapshot_before_compaction: false
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
index_interval: 128
encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra
=== ps ===
/usr/java/bin/java -ea -javaagent:/opt/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn1200M -XX:+HeapDumpOnOutOfMemoryError -Daccess.properties=/opt/cassandra/conf/access.properties -Dpasswd.properties=/opt/cassandra/conf/passwd.properties -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/opt/cassandra/log/cassandra.pid -cp /opt/cassandra/conf:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/antlr-3.2.jar:/opt/cassandra/lib/apache-cassandra-1.0.7.jar:/opt/cassandra/lib/apache-cassandra-clientutil-1.0.7.jar:/opt/cassandra/lib/apache-cassandra-thrift-1.0.7.jar:/opt/cassandra/lib/avro-1.4.0-fixes.jar:/opt/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.2.jar:/opt/cassandra/lib/commons-lang-2.4.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/cassandra/lib/guava-r08.jar:/opt/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/cassandra/lib/jamm-0.2.5.jar:/opt/cassandra/lib/jline-0.9.94.jar:/opt/cassandra/lib/jna.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/libthrift-0.6.jar:/opt/cassandra/lib/log4j-1.2.16.jar:/opt/cassandra/lib/mx4j-tools.jar:/opt/cassandra/lib/mx4j.jar:/opt/cassandra/lib/platform.jar:/opt/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/cassandra/lib/slf4j-api-1.6.1.jar:/opt/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/cassandra/lib/snakeyaml-1.6.jar:/opt/cassandra/lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
=== jmap -histo:live $PID | head -30 ===
num #instances #bytes class name
----------------------------------------------
1: 9368472 506649072 [B
2: 11820001 472800040 java.lang.ref.Finalizer
3: 11818288 472731520 org.apache.cassandra.cache.FreeableMemory
4: 9350412 448819776 java.nio.HeapByteBuffer
5: 9254312 370172480 java.math.BigInteger
6: 9259382 305187584 [I
7: 9320122 298243904 java.util.concurrent.ConcurrentHashMap$HashEntry
8: 9319917 298237344 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
9: 9349847 224396328 org.apache.cassandra.db.DecoratedKey
10: 9319917 223678008 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
11: 3396 213990608 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
12: 11819412 189110592 java.util.concurrent.atomic.AtomicInteger
13: 9253936 148062976 org.apache.cassandra.dht.BigIntegerToken
14: 2122 17298400 [Ljava.lang.Object;
15: 607125 14571000 java.lang.Long
16: 606820 14563680 org.apache.cassandra.utils.Pair
17: 345635 14312656 [C
18: 346425 11085600 java.lang.String
19: 963 10434360 [J
20: 43141 6139760 <constMethodKlass>
21: 43141 5877064 <methodKlass>
22: 4165 4706736 <constantPoolKlass>
23: 62828 3440624 <symbolKlass>
24: 4165 3128168 <instanceKlassKlass>
25: 3783 3015992 <constantPoolCacheKlass>
26: 3657 1779840 <methodDataKlass>
27: 34737 1111584 java.lang.ref.WeakReference
