DataStax Support Forums » Topic: Disks stats not available (workaround provided)

DataStax Support Forums » Topic: Disks stats not available (workaround provided) http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 08:12:38 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php nickmbailey on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-4476 Tue, 31 Jul 2012 21:43:59 +0000 nickmbailey 4476@http://www.datastax.com/support-forums/ Just as an update, the latest 2.1.2 release of OpsCenter will allow the agent to fail gracefully when encountering an NFS partition. Please let us know if you have any issues upgrading. thobbs on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2332 Fri, 22 Jun 2012 16:41:03 +0000 thobbs 2332@http://www.datastax.com/support-forums/ That's definitely something that's in our roadmap, but unfortunately, at the moment, we do just show an aggregate for all disks on all nodes in the cluster. I'll make a note in the ticket that we got a request for this. Anonymous on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2294 Fri, 22 Jun 2012 00:36:56 +0000 Anonymous 2294@http://www.datastax.com/support-forums/ Shouldn't opscenter only be showing (or trying to shows) IOPs for disks being used by Cassandra like the SS tables and commit logs? Why would opscenter show IOPs for non-related disks? thobbs on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2292 Thu, 21 Jun 2012 20:57:13 +0000 thobbs 2292@http://www.datastax.com/support-forums/ Ah, okay. The numbers you were seeing in 1.3.1 for IOPS were for your device mapper and sda devices, not NFS. When this bug is resolved, you should see that information again, but it still won't include NFS information. Anonymous on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2289 Thu, 21 Jun 2012 20:39:10 +0000 Anonymous 2289@http://www.datastax.com/support-forums/ On Opscenter 1.3.1 with NFS storage I can see storage capacity and disk IOPS. thobbs on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2236 Tue, 19 Jun 2012 17:38:43 +0000 thobbs 2236@http://www.datastax.com/support-forums/ Thanks, I appreciate the extra information. I'm assuming the dev mapper (looks like LVM) isn't related to the mounted NFS partition at all? When you say "opscenter was getting NFS disks stats for us on this ring in the 1.x release", are you just referring to it showing the amount of free vs used disk space, or are you referring to IOPS and disk utilization? Anonymous on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2235 Tue, 19 Jun 2012 16:19:49 +0000 Anonymous 2235@http://www.datastax.com/support-forums/ BTW, it opscenter was getting NFS disks stats for us on this ring in the 1.x release 1) stat --printf=%d /opt/cassandra_data 20 2) cat /proc/devices Character devices: 1 mem 4 /dev/vc/0 4 tty 4 ttyS 5 /dev/tty 5 /dev/console 5 /dev/ptmx 7 vcs 10 misc 13 input 21 sg 29 fb 99 ppdev 128 ptm 136 pts 162 raw 180 usb 189 usb_device 202 cpu/msr 203 cpu/cpuid 249 vmci 250 hidraw 251 usbmon 252 bsg 253 pcmcia 254 rtc Block devices: 1 ramdisk 259 blkext 7 loop 8 sd 9 md 65 sd 66 sd 67 sd 68 sd 69 sd 70 sd 71 sd 128 sd 129 sd 130 sd 131 sd 132 sd 133 sd 134 sd 135 sd 253 device-mapper 254 mdp 3) cat /proc/diskstats 1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 1 1 ram1 0 0 0 0 0 0 0 0 0 0 0 1 2 ram2 0 0 0 0 0 0 0 0 0 0 0 1 3 ram3 0 0 0 0 0 0 0 0 0 0 0 1 4 ram4 0 0 0 0 0 0 0 0 0 0 0 1 5 ram5 0 0 0 0 0 0 0 0 0 0 0 1 6 ram6 0 0 0 0 0 0 0 0 0 0 0 1 7 ram7 0 0 0 0 0 0 0 0 0 0 0 1 8 ram8 0 0 0 0 0 0 0 0 0 0 0 1 9 ram9 0 0 0 0 0 0 0 0 0 0 0 1 10 ram10 0 0 0 0 0 0 0 0 0 0 0 1 11 ram11 0 0 0 0 0 0 0 0 0 0 0 1 12 ram12 0 0 0 0 0 0 0 0 0 0 0 1 13 ram13 0 0 0 0 0 0 0 0 0 0 0 1 14 ram14 0 0 0 0 0 0 0 0 0 0 0 1 15 ram15 0 0 0 0 0 0 0 0 0 0 0 7 0 loop0 0 0 0 0 0 0 0 0 0 0 0 7 1 loop1 0 0 0 0 0 0 0 0 0 0 0 7 2 loop2 0 0 0 0 0 0 0 0 0 0 0 7 3 loop3 0 0 0 0 0 0 0 0 0 0 0 7 4 loop4 0 0 0 0 0 0 0 0 0 0 0 7 5 loop5 0 0 0 0 0 0 0 0 0 0 0 7 6 loop6 0 0 0 0 0 0 0 0 0 0 0 7 7 loop7 0 0 0 0 0 0 0 0 0 0 0 8 0 sda 79123 52951 12081686 176969 26277300 201488089 1822135354 212432222 0 16358669 212599753 8 1 sda1 1716 293 24436 1148 5 0 34 1 0 1044 1149 8 2 sda2 77224 52658 12055786 175612 26277295 201488089 1822135320 212432221 0 16357445 212598396 253 0 dm-0 37781 0 827388 109115 692479 0 5539832 110153915 0 105746 110263028 253 1 dm-1 2703 0 21624 4006 3100 0 24800 23829 0 1333 27835 253 2 dm-2 25356 0 365234 78572 164952677 0 1319621416 313941387 0 5347648 314483375 253 3 dm-3 9446 0 75626 25768 6828344 0 54626752 13095976 0 3160995 13142550 253 4 dm-4 52105 0 10741194 111294 55290315 0 442322520 539514314 0 8360645 539788627 4) iostat -xd Linux 2.6.32-220.el6.x86_64 (ccpcas-dt-b101-q.THE_DOMAIN) 06/19/2012 _x86_64_ (4 CPU) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 18.21 0.01 2.38 1.09 164.70 69.59 0.02 8.07 0.62 0.15 dm-0 0.00 0.00 0.00 0.06 0.07 0.50 8.72 0.01 150.99 0.14 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 4.80 0.23 0.00 dm-2 0.00 0.00 0.00 14.91 0.03 119.28 8.00 0.03 1.90 0.03 0.05 dm-3 0.00 0.00 0.00 0.62 0.01 4.94 8.00 0.00 1.92 0.46 0.03 dm-4 0.00 0.00 0.00 5.00 0.97 39.98 8.19 0.05 9.75 0.15 0.08 thobbs on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2215 Mon, 18 Jun 2012 16:48:00 +0000 thobbs 2215@http://www.datastax.com/support-forums/ Thanks for the additional info. It looks like the NFS device won't work with the current version of the agent, but if I can get a bit more information from you, we might be able to support it in a future release. Can you post: 1) "<code>stat --printf=%d /opt/cassandra_data</code>" 2) "<code>cat /proc/devices</code>" 3) "<code>cat /proc/diskstats</code>" 4) "<code>iostat -xd</code>" Thanks, much appreciated. Anonymous on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2214 Mon, 18 Jun 2012 16:29:42 +0000 Anonymous 2214@http://www.datastax.com/support-forums/ Note that in this ring we are using NFS for storage (in ghetto mode waiting for the SSDs to arrive) 1) Centos 6.2 x86_64 2) df /opt/cassandra_data Filesystem 1K-blocks Used Available Use% Mounted on 192.168.8.128:/vol/cas1vol003/sst_node01 209715200 19330624 190384576 10% /opt/cassandra_data 3) stat --printf=%D /opt/cassandra_data 14 4) cat /proc/partitions major minor #blocks name 8 0 78643200 sda 8 1 104416 sda1 8 2 78538752 sda2 253 0 10485760 dm-0 253 1 1048576 dm-1 253 2 31457280 dm-2 253 3 2097152 dm-3 253 4 33423360 dm-4 nickmbailey on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2213 Mon, 18 Jun 2012 15:29:40 +0000 nickmbailey 2213@http://www.datastax.com/support-forums/ This is actually a problem trying to determine the device/partitions for your data directories. A couple of questions: 1) What operating system/distribution/version are you running? 2) What is the output of 'df <dir>' for each directory configured as a data directory in your cassandra.yaml? 3) What is the output of 'stat --printf=%D <dir>' for each directory configured as a data directory in your cassandra.yaml? 4) What is the output of 'cat /proc/partitions'? Anonymous on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2196 Fri, 15 Jun 2012 23:40:02 +0000 Anonymous 2196@http://www.datastax.com/support-forums/ I'm having a similar problem in 2.0 where storage info isn't reported, but the exception is a little different. echo $LANG en_US.UTF-8 What command is opscenter trying to use? I can run iostat fine. ERROR [pool-3-thread-3] 2012-06-12 03:53:33,105 Error getting devices and partitions java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "" at clojure.lang.LazySeq.sval(LazySeq.java:47) at clojure.lang.LazySeq.seq(LazySeq.java:56) at clojure.lang.LazySeq.cons(LazySeq.java:99) at clojure.lang.LazySeq.cons(LazySeq.java:17) at clojure.lang.RT.conj(RT.java:524) at clojure.core$conj.invoke(core.clj:78) at clojure.core$conj.doInvoke(core.clj:81) at clojure.lang.RestFn.invoke(RestFn.java:442) at opsagent.nodedetails$get_disk_info.invoke(nodedetails.clj:86) at opsagent.nodedetails$run_longtime_collection$fn__2803.invoke(nodedetails.clj:167) nickmbailey on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-2151 Wed, 13 Jun 2012 14:49:08 +0000 nickmbailey 2151@http://www.datastax.com/support-forums/ Just as an update. I wanted to let you know that the latest release of OpsCenter (2.1) correctly sets the locale setting when starting the agent process. This should prevent the need of changing any locale settings manually. nickmbailey on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided/page/2#post-1203 Wed, 22 Feb 2012 15:45:54 +0000 nickmbailey 1203@http://www.datastax.com/support-forums/ Thanks for the bug report. We've filed a bug to fix this problem. Just to be clear, you should only need to change the locale setting on your cassandra machines where the agent is running. You do not need to change it on the main OpsCenter machine. secu on "Disks stats not available (workaround provided)" http://www.datastax.com/support-forums/topic/disks-stats-not-available-workaround-provided#post-1202 Wed, 22 Feb 2012 15:21:15 +0000 secu 1202@http://www.datastax.com/support-forums/ Hi, In OpsCenter 1.4 disks stats are not available if numeric separator is ",". Depending on system locale -- e.g. fr_FR, it_IT, de_DE -- iostat's output contains commas. In such a case, agents throw a NumberFormatException (see below for the traceback). FYI a workaround is to unset LANG in bashrc : On OpsCenter server : <code> # su - opscenter -bash-3.2$ echo "export LANG=" > .bashrc -bash-3.2$ ln -s .bashrc .bash_profile </code> For each agent : <code> # su - opscenter-agent -bash-3.2$ echo "export LANG=" > .bashrc -bash-3.2$ ln -s .bashrc .bash_profile </code> Complete stacktrace found in agent.log : <code> ERROR [pool-4-thread-7] 2012-02-22 14:24:46,945 Long os-stats collector failed java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "0,00" at clojure.lang.LazySeq.sval(LazySeq.java:47) at clojure.lang.LazySeq.seq(LazySeq.java:56) at clojure.lang.RT.seq(RT.java:450) at clojure.core$seq.invoke(core.clj:122) at clojure.core$r.invoke(core.clj:793) at clojure.core$into.invoke(core.clj:2614) at opsagent.os.linux_metrics$parse_iostat_line.invoke(linux_metrics.clj:190) at opsagent.os.linux_metrics$parse_iostats.invoke(linux_metrics.clj:215) at opsagent.proc$parse_results.invoke(proc.clj:100) at opsagent.proc$parse_output.doInvoke(proc.clj:114) at clojure.lang.RestFn.invoke(RestFn.java:442) at opsagent.os.linux_metrics$collectors$proc_runner__559.invoke(linux_metrics.clj:244) at opsagent.os.linux_metrics$get_iostats.invoke(linux_metrics.clj:220) at opsagent.os.linux_metrics$collectors$wrap_long_collector__563$fn__564.invoke(linux_metrics.clj:249) at opsagent.util$error_wrap.doInvoke(util.clj:97) at clojure.lang.RestFn.invoke(RestFn.java:423) at opsagent.os.collection$start_pool$fn__818.invoke(collection.clj:45) at clojure.lang.AFn.run(AFn.java:24) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "0,00" at clojure.lang.LazySeq.sval(LazySeq.java:47) at clojure.lang.LazySeq.seq(LazySeq.java:56) at clojure.lang.RT.seq(RT.java:450) at clojure.core$seq.invoke(core.clj:122) at clojure.core$map$fn__429.invoke(core.clj:2099) at clojure.lang.LazySeq.sval(LazySeq.java:42) ... 26 more Caused by: java.lang.NumberFormatException: For input string: "0,00" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224) at java.lang.Float.parseFloat(Float.java:422) at opsagent.os.linux_metrics$parse_iostat_line$fn__548.invoke(linux_metrics.clj:189) at clojure.core$map$fn__425.invoke(core.clj:2096) at clojure.lang.LazySeq.sval(LazySeq.java:42) ... 31 more </code>