Seeing memory swap issue on cassandra nodes

(4 posts) (2 voices)

• Started 5 months ago by srk_user
• Latest reply from Sven
• This topic is not resolved

Tags:

No tags yet.

1. 

   srk_user
   Member

   We are seeing memory swap issues on cassandra nodes. We are running DataStax Enterprise 2.1.1. Each node with 6CPU 16GB memory.

   The heap is setup as follows

   MAX_HEAP_SIZE="8G"
   HEAP_NEWSIZE="800M"

   Here's what we are seeing on the server.. Any idea why we are seeing the swap issue and how can we resolve it

   top - 09:43:06 up 36 days, 19:25, 1 user, load average: 0.00, 0.03, 0.02
   Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie
   Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 96.6%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Mem: 16435996k total, 15833124k used, 602872k free, 465808k buffers
   Swap: 2068400k total, 2068356k used, 44k free, 5466128k cached

   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   4479 casadmin 23 0 268g 12g 3.9g S 0.7 78.8 5595:34 /usr/java/jre1.6.0_29/bin/java -ea -javaagent:/opt/nosql/dse-2.1.1/resources/cassandra/lib/jamm-0.2.5.jar -
   21326 sr88197 15 0 12756 1180 844 R 0.7 0.0 0:00.01 top
   31548 casadmin 18 0 507m 114m 9580 S 0.7 0.7 184:30.81 /usr/java/jre1.6.0_29/bin/java -Xmx40M -Xms40M -Djavax.net.ssl.trustStore=ssl/agentKeyStore -Djavax.net.ssl

   Posted 5 months ago #
2. 

   srk_user
   Member

   I changed the MAX_HEAP_SIZE to "4G" and restarted the nodes. On Opscenter, when I check the OS:memory for each node, I see 4GB of used memory but cached memory is over 7GB and seems to growing.. My question is how do I control the cached memory?

   Posted 5 months ago #
3. 

   srk_user
   Member

   Any update on this?

   Posted 4 months ago #
4. 

   Sven
   Administrator

   Has reducing the max heap size resolved the swapping issue?

   Keep in mind that C* uses off heap memory as well to store cache data. You may want to look into tuning that, see here http://www.datastax.com/docs/1.1/operations/tuning for more detail.

   With respect to the high memory usage reported for the Cassandra process, check here: http://wiki.apache.org/cassandra/FAQ#mmap.

   With respect to the high cached value. That is a Linux'ism, it is the OS level cache for file reads (see here for more information: http://egloo.wordpress.com/2008/10/29/linux-cached-memory/). It should not interfere with normal operations as it will only use unused memory. To find out what processes are actually using swap (in case that is still an issue after tuning the JVM and C*), check here http://stackoverflow.com/questions/479953/how-to-find-out-which-processes-are-swapping-in-linux. I would expect that to be C*, but with some tuning you should be able to reduce swap use.

   Thanks,
   Sven

   Posted 4 months ago #

RSS feed for this topic

Reply

You must log in to post.