Cassandra freezed for a few hours with 100% CPU

(8 posts) (3 voices)

• Started 1 year ago by
• Latest reply from
• This topic is not a support question

Tags:

• freeze
• freezing

1. 

   Anonymous
   Unregistered

   Hello.
   Confused with the problem when, after the few success batch operations, on next batch operation Cassandra hung and become unresponseble. This has been going on for several hours, i provide munin graph of memory usage and load CPU. Disk operations is not happening. No fail records in log. There are no any(!) records in log after freeze happend.

   top command says this:
   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   1362 cassandr 20 0 10.5g 8.5g 15m S 100 27.1 75:18.82 jsvc
   11080 root 20 0 21468 1348 992 R 0 0.0 0:00.01 top
   1 root 20 0 24060 2168 1320 S 0 0.0 0:00.74 init

   Munin graps since freeze started:
   CPU: http://i5.pixs.ru/storage/4/3/6/cpudaypng_5296647_4572436.png
   MEM: http://i5.pixs.ru/storage/4/3/8/memorydayp_6892077_4572438.png

   I have no idea what to do with this :(

   Posted 1 year ago #
2. 

   Anonymous
   Unregistered

   Sorry, installation info:

   Package: cassandra
   Status: install ok installed
   Priority: extra
   Section: misc
   Installed-Size: 10084
   Maintainer: Eric Evans <eevans@apache.org>
   Architecture: all
   Version: 1.0.9
   Replaces: apache-cassandra1
   Depends: openjdk-6-jre-headless (>= 6b11) | java6-runtime, jsvc (>= 1.0), libcommons-daemon-java (>= 1.0), adduser, libjna-java
   Conflicts: apache-cassandra1
   Conffiles:
   /etc/security/limits.d/cassandra.conf 97e54ecf9431b60336f4bc066a62908b
   /etc/init.d/cassandra 79d6d592e20d62c488a18330b8ed5de7
   /etc/sysctl.d/cassandra.conf db94c1e32cd88ebc1fd840b2c091ac3f
   /etc/cassandra/cassandra-env.sh 6440ea9539f0bceb500a3809606c19e4
   /etc/cassandra/log4j-server.properties 3fa812af246b2a29845c73b8456ea97e
   /etc/cassandra/cassandra.yaml ee75ec8f259f675e18f8ef4b80dfc877
   /etc/default/cassandra 98a0a8e9283cc085b8eddee9da28f778
   Description: distributed storage system for structured data
   Cassandra is a distributed (peer-to-peer) system for the management
   and storage of structured data.
   Homepage: http://cassandra.apache.org

   Running on:
   Distributor ID: Ubuntu
   Description: Ubuntu 11.10
   Release: 11.10
   Codename: oneiric

   Linux jopa 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   I also had this records in log:

   Memtable.java (line 151) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead

   don't know if it releated with the problem.

   Posted 1 year ago #
4. 

   thepaul
   Moderator

   Are you using the Sun JRE? That may be part of the issue.

   Otherwise, your munin graphs may help us solve the problem. Was the freeze event at around 03:30 or 12:00? Both of those times seem to correlate to big changes in the graphs.

   Also, are there any other performance data you can send? Iostat output, CPU load, disk iops, disk bw, process forks- not sure yet what might help.

   Posted 1 year ago #
5. 

   Anonymous
   Unregistered

   I'm actually a C++ developer, and my knowledge of java is very very basic. Don't know wich kind oа JRE i am using, but i installed cassandra with apt-get install without(!) any other additional steps, so i think it shoulde be working with default configuration and with default JRE. (How to know wich JRE cassi does use ?)
   I also read this ticket: https://issues.apache.org/jira/browse/CASSANDRA-2845
   As i said, i am very basic in java, so all i did after reading this ticket - restarted cassi with "cassandra -f" under the screen, and now it works ok... don't know if problem is really gone.... not sure...

   About graphs.
   Freezing began in the 12th and the end of the graph. For that would give you more information, I packed all the graphics that munin I have for this machine to the archive and put it here:
   http://narod.ru/disk/46525796001.e240def8d4b20e7a3813fcbad8f6fed7/munin_cassis.rar.html

   On this new pictures "freezing zone" moved to the left side of the graphs, i made a snapshot of graphs just now.

   PS: On the new graphs you can see another zone with hi CPU useage (with peaks) - its ok, here i'm runnign a script wich inflating db with big batch_mutate calls.

   And thank you for fast response!

   Posted 1 year ago #
6. 

   Anonymous
   Unregistered

   In addition to "wich JRE i'm using" - on this machine i have only postgresql db and cassandra installed. and gcc tools for compiling my c++ stuff. Nothing else there. So i did not installed any thirdparty JRE...

   Posted 1 year ago #
7. 

   joaquin
   Administrator

   Hello Andre,

   Due to certain licensing agreements, OpenJDK is always installed by default when running apt-get. Here are the instructions to get the official Sun Java JVM installed:
   http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb

   Once you run these commands, perform a rolling restart and run ps auwx | grep cass and you should be able to see the JVM in the first portion of that output.

   Do let us know if this fixes your problem.

   Thanks,
   Joaquin

   Posted 1 year ago #
8. 

   Anonymous
   Unregistered

   Hi!
   I did all steps described here:
   http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb
   and additionaly run this:
   root@jopa:~# export JAVA_HOME=/opt/java/64/jdk1.6.0_30
   root@jopa:~# export PATH=$JAVA_HOME/bin:$PATH
   root@jopa:~# export
   declare -x HOME="/root"
   declare -x JAVA_HOME="/opt/java/64/jdk1.6.0_30"
   declare -x LANG="en_US.UTF-8"
   declare -x LANGUAGE="en_US:en"
   declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
   declare -x LESSOPEN="| /usr/bin/lesspipe %s"
   declare -x LOGNAME="root"
   declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:"
   declare -x MAIL="/var/mail/root"
   declare -x OLDPWD
   declare -x PATH="/opt/java/64/jdk1.6.0_30/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
   declare -x PWD="/root"
   declare -x SHELL="/bin/bash"
   declare -x SHLVL="1"
   declare -x SSH_CLIENT="192.168.1.105 58688 22"
   declare -x SSH_CONNECTION="192.168.1.105 58688 192.168.1.102 22"
   declare -x SSH_TTY="/dev/pts/0"
   declare -x TERM="xterm"
   declare -x USER="root"

   After this i restarted machine and with command "ps auwx | grep cass" got this output:
   root@jopa:~# ps auwx | grep cass
   root 1357 0.0 0.0 18940 400 ? Ss 13:45 0:00 jsvc.exec -user cassandra -home /opt/java/64/jdk1.6.0_30/bin/../ -pidfile /var/run/cassandra.pid -errfi
   le &1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.
   0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share
   /cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-
   scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/u
   sr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.ja
   r:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassand
   ra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.9.jar:/usr/share/cassandra/apache-cassandra-thrift-1
   .0.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properti
   es -XX:HeapDumpPath=/var/lib/cassandra/java_1334915126.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1334915126.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8039M -Xmx8039M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIP
   v4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon
   109 1358 80.1 26.2 9160232 8629500 ? SLl 13:45 0:12 jsvc.exec -user cassandra -home /opt/java/64/jdk1.6.0_30/bin/../ -pidfile /var/run/cassandra.pid -errfi
   le &1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.
   0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share
   /cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-
   scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/u
   sr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.ja
   r:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassand
   ra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.9.jar:/usr/share/cassandra/apache-cassandra-thrift-1
   .0.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properti
   es -XX:HeapDumpPath=/var/lib/cassandra/java_1334915126.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1334915126.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8039M -Xmx8039M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIP
   v4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon
   root 1519 0.0 0.0 6264 188 pts/0 R+ 13:45 0:00 grep --color=auto cass

   Also getting version give this:
   root@jopa:~# java -version
   java version "1.6.0_30"
   Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
   Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

   I can't see JVM the first portion of that output so i'm still don't know with runtime cassandra use.
   At this time i do some developing and don't use cassandra db in any load...i'll start using it in two days aprox... and say if problem is gone.
   Anyway - BIG THANKS for you help!

   Posted 1 year ago #

RSS feed for this topic