DataStax Support Forums » Topic: Cassandra freezed for a few hours with 100% CPU

DataStax Support Forums » Topic: Cassandra freezed for a few hours with 100% CPU http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 05:43:49 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1726 Fri, 20 Apr 2012 10:20:06 +0000 Anonymous 1726@http://www.datastax.com/support-forums/ Hi! I did all steps described here: <a href="http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb" rel="nofollow">http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb</a> and additionaly run this: root@jopa:~# export JAVA_HOME=/opt/java/64/jdk1.6.0_30 root@jopa:~# export PATH=$JAVA_HOME/bin:$PATH root@jopa:~# export declare -x HOME="/root" declare -x JAVA_HOME="/opt/java/64/jdk1.6.0_30" declare -x LANG="en_US.UTF-8" declare -x LANGUAGE="en_US:en" declare -x LESSCLOSE="/usr/bin/lesspipe %s %s" declare -x LESSOPEN="| /usr/bin/lesspipe %s" declare -x LOGNAME="root" declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:" declare -x MAIL="/var/mail/root" declare -x OLDPWD declare -x PATH="/opt/java/64/jdk1.6.0_30/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" declare -x PWD="/root" declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x SSH_CLIENT="192.168.1.105 58688 22" declare -x SSH_CONNECTION="192.168.1.105 58688 192.168.1.102 22" declare -x SSH_TTY="/dev/pts/0" declare -x TERM="xterm" declare -x USER="root" After this i restarted machine and with command "ps auwx | grep cass" got this output: root@jopa:~# ps auwx | grep cass root 1357 0.0 0.0 18940 400 ? Ss 13:45 0:00 jsvc.exec -user cassandra -home /opt/java/64/jdk1.6.0_30/bin/../ -pidfile /var/run/cassandra.pid -errfi le &1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4. 0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share /cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high- scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/u sr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.ja r:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassand ra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.9.jar:/usr/share/cassandra/apache-cassandra-thrift-1 .0.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properti es -XX:HeapDumpPath=/var/lib/cassandra/java_1334915126.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1334915126.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8039M -Xmx8039M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIP v4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon 109 1358 80.1 26.2 9160232 8629500 ? SLl 13:45 0:12 jsvc.exec -user cassandra -home /opt/java/64/jdk1.6.0_30/bin/../ -pidfile /var/run/cassandra.pid -errfi le &1 -outfile /var/log/cassandra/output.log -cp /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4. 0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share /cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high- scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/u sr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.ja r:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassand ra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.9.jar:/usr/share/cassandra/apache-cassandra-thrift-1 .0.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properti es -XX:HeapDumpPath=/var/lib/cassandra/java_1334915126.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1334915126.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8039M -Xmx8039M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIP v4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.thrift.CassandraDaemon root 1519 0.0 0.0 6264 188 pts/0 R+ 13:45 0:00 grep --color=auto cass Also getting version give this: root@jopa:~# java -version java version "1.6.0_30" Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) I can't see JVM the first portion of that output so i'm still don't know with runtime cassandra use. At this time i do some developing and don't use cassandra db in any load...i'll start using it in two days aprox... and say if problem is gone. Anyway - BIG THANKS for you help! joaquin on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1691 Wed, 18 Apr 2012 18:09:45 +0000 joaquin 1691@http://www.datastax.com/support-forums/ Hello Andre, Due to certain licensing agreements, OpenJDK is always installed by default when running apt-get. Here are the instructions to get the official Sun Java JVM installed: <a href="http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb" rel="nofollow">http://www.datastax.com/docs/1.0/install/install_package#install-jre-deb</a> Once you run these commands, perform a rolling restart and run <code>ps auwx | grep cass</code> and you should be able to see the JVM in the first portion of that output. Do let us know if this fixes your problem. Thanks, Joaquin Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1689 Wed, 18 Apr 2012 17:53:25 +0000 Anonymous 1689@http://www.datastax.com/support-forums/ In addition to "wich JRE i'm using" - on this machine i have only postgresql db and cassandra installed. and gcc tools for compiling my c++ stuff. Nothing else there. So i did not installed any thirdparty JRE... Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1685 Wed, 18 Apr 2012 17:48:42 +0000 Anonymous 1685@http://www.datastax.com/support-forums/ I'm actually a C++ developer, and my knowledge of java is very very basic. Don't know wich kind oа JRE i am using, but i installed cassandra with apt-get install without(!) any other additional steps, so i think it shoulde be working with default configuration and with default JRE. (How to know wich JRE cassi does use ?) I also read this ticket: <a href="https://issues.apache.org/jira/browse/CASSANDRA-2845" rel="nofollow">https://issues.apache.org/jira/browse/CASSANDRA-2845</a> As i said, i am very basic in java, so all i did after reading this ticket - restarted cassi with "cassandra -f" under the screen, and now it works ok... don't know if problem is really gone.... not sure... About graphs. Freezing began in the 12th and the end of the graph. For that would give you more information, I packed all the graphics that munin I have for this machine to the archive and put it here: <a href="http://narod.ru/disk/46525796001.e240def8d4b20e7a3813fcbad8f6fed7/munin_cassis.rar.html" rel="nofollow">http://narod.ru/disk/46525796001.e240def8d4b20e7a3813fcbad8f6fed7/munin_cassis.rar.html</a> On this new pictures "freezing zone" moved to the left side of the graphs, i made a snapshot of graphs just now. PS: On the new graphs you can see another zone with hi CPU useage (with peaks) - its ok, here i'm runnign a script wich inflating db with big batch_mutate calls. And thank you for fast response! thepaul on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1664 Tue, 17 Apr 2012 21:25:34 +0000 thepaul 1664@http://www.datastax.com/support-forums/ Are you using the Sun JRE? That may be part of the issue. Otherwise, your munin graphs may help us solve the problem. Was the freeze event at around 03:30 or 12:00? Both of those times seem to correlate to big changes in the graphs. Also, are there any other performance data you can send? Iostat output, CPU load, disk iops, disk bw, process forks- not sure yet what might help. Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1650 Tue, 17 Apr 2012 14:36:55 +0000 Anonymous 1650@http://www.datastax.com/support-forums/ I also had this records in log: Memtable.java (line 151) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead don't know if it releated with the problem. Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1648 Tue, 17 Apr 2012 12:38:46 +0000 Anonymous 1648@http://www.datastax.com/support-forums/ Sorry, installation info: Package: cassandra Status: install ok installed Priority: extra Section: misc Installed-Size: 10084 Maintainer: Eric Evans <eevans@apache.org> Architecture: all Version: 1.0.9 Replaces: apache-cassandra1 Depends: openjdk-6-jre-headless (>= 6b11) | java6-runtime, jsvc (>= 1.0), libcommons-daemon-java (>= 1.0), adduser, libjna-java Conflicts: apache-cassandra1 Conffiles: /etc/security/limits.d/cassandra.conf 97e54ecf9431b60336f4bc066a62908b /etc/init.d/cassandra 79d6d592e20d62c488a18330b8ed5de7 /etc/sysctl.d/cassandra.conf db94c1e32cd88ebc1fd840b2c091ac3f /etc/cassandra/cassandra-env.sh 6440ea9539f0bceb500a3809606c19e4 /etc/cassandra/log4j-server.properties 3fa812af246b2a29845c73b8456ea97e /etc/cassandra/cassandra.yaml ee75ec8f259f675e18f8ef4b80dfc877 /etc/default/cassandra 98a0a8e9283cc085b8eddee9da28f778 Description: distributed storage system for structured data Cassandra is a distributed (peer-to-peer) system for the management and storage of structured data. Homepage: <a href="http://cassandra.apache.org" rel="nofollow">http://cassandra.apache.org</a> Running on: Distributor ID: Ubuntu Description: Ubuntu 11.10 Release: 11.10 Codename: oneiric Linux jopa 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Anonymous on "Cassandra freezed for a few hours with 100% CPU" http://www.datastax.com/support-forums/topic/cassandra-freezed-for-a-few-hours-with-100-cpu#post-1647 Tue, 17 Apr 2012 12:33:04 +0000 Anonymous 1647@http://www.datastax.com/support-forums/ Hello. Confused with the problem when, after the few success batch operations, on next batch operation Cassandra hung and become unresponseble. This has been going on for several hours, i provide munin graph of memory usage and load CPU. Disk operations is not happening. No fail records in log. There are no any(!) records in log after freeze happend. top command says this: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1362 cassandr 20 0 10.5g 8.5g 15m S 100 27.1 75:18.82 jsvc 11080 root 20 0 21468 1348 992 R 0 0.0 0:00.01 top 1 root 20 0 24060 2168 1320 S 0 0.0 0:00.74 init Munin graps since freeze started: CPU: <a href="http://i5.pixs.ru/storage/4/3/6/cpudaypng_5296647_4572436.png" rel="nofollow">http://i5.pixs.ru/storage/4/3/6/cpudaypng_5296647_4572436.png</a> MEM: <a href="http://i5.pixs.ru/storage/4/3/8/memorydayp_6892077_4572438.png" rel="nofollow">http://i5.pixs.ru/storage/4/3/8/memorydayp_6892077_4572438.png</a> I have no idea what to do with this :(