Disk usage not working

(8 posts) (3 voices)

• Started 3 months ago by daniel
• Latest reply from thobbs
• This topic is not a support question

Tags:

No tags yet.

1. 

   daniel
   Member

   I just installed OpsCenter and connected it to a new Cluster running Cassandra 1.2.1. OpsCenter was installed from the apt repository and is up-to-date. I can see most of the graphs except for disk usage, disk throughput, etc which will read "NO DATA". I didn't see any errors in the logs that looked related. Any ideas?

   root@cassandra-dev2:~# uname -a
   Linux cassandra-dev2 2.6.32-33-server #72-Ubuntu SMP Fri Jul 29 21:21:55 UTC 2011 x86_64 GNU/Linux
   root@cassandra-dev2:~# df /var/lib/cassandra/data
   Filesystem 1K-blocks Used Available Use% Mounted on
   /dev/vda1 33027920 25808100 5542256 83% /
   root@cassandra-dev2:~# stat --printf=%D /var/lib/cassandra/data
   fc01root@cassandra-dev2:~# cat /proc/partitions
   major minor #blocks name

   252 0 33554432 vda
   252 1 33553376 vda1
   252 16 2097152 vdb
   252 17 2097024 vdb1
   root@cassandra-dev2:~# cat /proc/devices
   Character devices:
   1 mem
   4 /dev/vc/0
   4 tty
   4 ttyS
   5 /dev/tty
   5 /dev/console
   5 /dev/ptmx
   6 lp
   7 vcs
   10 misc
   13 input
   21 sg
   29 fb
   108 ppp
   128 ptm
   136 pts
   180 usb
   189 usb_device
   252 usbmon
   253 bsg
   254 rtc

   Block devices:
   1 ramdisk
   2 fd
   259 blkext
   7 loop
   8 sd
   9 md
   11 sr
   65 sd
   66 sd
   67 sd
   68 sd
   69 sd
   70 sd
   71 sd
   128 sd
   129 sd
   130 sd
   131 sd
   132 sd
   133 sd
   134 sd
   135 sd
   251 device-mapper
   252 virtblk
   253 pktcdvd
   254 mdp
   root@cassandra-dev2:~# cat /proc/diskstats
   1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
   1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
   1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
   1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
   1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
   1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
   1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
   1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
   1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
   1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
   1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
   1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
   1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
   1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
   1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
   1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
   7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
   7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
   7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
   7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
   7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
   7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
   7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
   7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
   252 0 vda 534991 11464 125784114 414080 1371462 36624133 303968336 6568120 0 2465000 6981020
   252 1 vda1 534952 11464 125783802 414050 1371462 36624133 303968336 6568120 0 2464990 6980990
   252 16 vdb 583 994 9473 970 610 5524 49072 6130 0 6960 7090
   252 17 vdb1 560 994 9289 940 610 5524 49072 6130 0 6930 7060
   root@cassandra-dev2:~# iostat -xd
   Linux 2.6.32-33-server (cassandra-dev2) 03/04/2013 _x86_64_ (2 CPU)

   Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
   vda 0.01 24.85 0.36 0.93 85.35 206.25 225.42 0.00 3.66 1.29 0.17
   vdb 0.00 0.00 0.00 0.00 0.01 0.03 49.07 0.00 5.95 5.83 0.00

   root@cassandra-dev2:~#

   Posted 3 months ago #
2. 

   tupton
   Moderator

   Are the agents installed, running, and connected on your nodes?

   If so, are there any errors at all in the agent logs? You said you didn't see any that were related, but were there any at all?

   Posted 3 months ago #
3. 

   daniel
   Member

   There were errors when I initially set up the cluster about not being able to connect because the agents were not running yet. After I installed the agents correctly, those disappeared. All agents are currently connected to OpsCenter. They are currently reporting other data like CPU usage, reads/sec, writes/sec, etc.

   Test Cluster
   (Cassandra 1.2.1)
   |
   All agents connected
   Datacenter dc1
   3 nodes
   Datacenter dc2
   3 nodes

   This did just show up on the OpsCenter node, which does look related:
   2013-03-04 17:10:11-0500 [] ERROR: There was an error while generating a cluster report. The data partitions for node 10.60.4.239 have not been determined
   2013-03-04 17:10:11-0500 [] ERROR: Error generating the cluster report: The data partitions for node 10.60.4.239 have not been determined

   Here's the errors on one of the Cassandra nodes:

   root@cassandra-dev27:/var/log/cassandra# grep 'ERROR' system.log | cut -d ' ' -f 5- | sort | uniq -c | sort -nr
   32048 CommitLogSegment.java (line 194) Attempted to write commit log entry for unrecognized column family: 366983a6-c448-3094-82a9-9facc5200bbf
   18600 Table.java (line 393) Attempting to mutate non-existant column family 366983a6-c448-3094-82a9-9facc5200bbf
   1176 ErrorMessage.java (line 197) Unexpected exception during request
   73 TNonblockingServer.java (line 468) Read an invalid frame size of 0. Are you using TFramedTransport on the client side?
   70 TNonblockingServer.java (line 638) Unexpected exception while invoking!
   6 OutboundTcpConnection.java (line 196) error writing to /10.60.9.0
   1 CassandraDaemon.java (line 181) Fatal exception during initialization

   root@cassandra-dev27:/var/log/cassandra# grep 'ERROR' output.log | cut -d ' ' -f 3- | sort | uniq -c | sort -nr
   32048 Attempted to write commit log entry for unrecognized column family: 366983a6-c448-3094-82a9-9facc5200bbf
   18600 Attempting to mutate non-existant column family 366983a6-c448-3094-82a9-9facc5200bbf
   1176 Unexpected exception during request
   73 Read an invalid frame size of 0. Are you using TFramedTransport on the client side?
   70 Unexpected exception while invoking!
   6 error writing to /10.60.9.0
   1 Fatal exception during initialization
   root@cassandra-dev27:/var/log/cassandra#

   Posted 3 months ago #
4. 

   thobbs
   Administrator

   Daniel,

   Thanks for the info so far, it's very helpful.

   Can I get the output of 'df /var/lib/cassandra/commitlog' and 'stat --printf=%D /var/lib/cassandra/commitlog', assuming your commitlog is stored there?

   Last, can I get the output of 'df --print-type'?

   Thanks again.

   Posted 3 months ago #
5. 

   daniel
   Member

   root@cassandra-dev2:~# df /var/lib/cassandra/commitlog
   Filesystem 1K-blocks Used Available Use% Mounted on
   /dev/vda1 33027920 26015460 5334896 83% /
   root@cassandra-dev2:~# stat --printf=%D /var/lib/cassandra/commitlog
   fc01root@cassandra-dev2:~# df --print-type
   Filesystem Type 1K-blocks Used Available Use% Mounted on
   /dev/vda1 ext4 33027920 26015460 5334896 83% /
   none devtmpfs 2024496 164 2024332 1% /dev
   none tmpfs 2028944 0 2028944 0% /dev/shm
   none tmpfs 2028944 68 2028876 1% /var/run
   none tmpfs 2028944 0 2028944 0% /var/lock
   none tmpfs 2028944 0 2028944 0% /lib/init/rw
   none debugfs 33027920 26015460 5334896 83% /var/lib/ureadahead/debugfs
   root@cassandra-dev2:~#

   Posted 3 months ago #
6. 

   thobbs
   Administrator

   Hmmm, it seems like everything should be working on your system, so I'm not sure why you're not getting stats. Can you try enabling debug logging on one of the agents (edit /etc/opscenter-agent/log4j.properties and switch INFO to DEBUG if the agent was installed through the UI or a package; conf/log4j.properties otherwise), restarting it, and pasting the first minute of logs?

   Posted 3 months ago #
7. 

   daniel
   Member

   I was looking in the Cassandra node log for errors instead of the agent log. I was not aware they were in different places. It turned out to be a permissions issue. I was seeing these errors:
   ERROR [Initialization] 2013-03-04 11:44:53,905 Error getting devices and partitions
   java.io.IOException: Process failed: df /var/lib/cassandra/commitlog
   Exit val: 1
   Output:
   df: `/var/lib/cassandra/commitlog': Permission denied

   ERROR [pool-5-thread-3] 2013-03-07 09:14:56,739 Error getting devices and partitions
   java.io.IOException: Process failed: df /var/lib/cassandra/saved_caches
   Exit val: 1
   Output:
   df: `/var/lib/cassandra/saved_caches': Permission denied

   ERROR [Initialization] 2013-03-07 09:15:10,719 Error getting devices and partitions
   java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Process failed: df /var/lib/cassandra/data
   Exit val: 1
   Output:
   df: `/var/lib/cassandra/data': Permission denied

   These files had permissions of 700 with the user cassandra and group cassandra. The agent runs under user opscenter-agent who is in the group cassandra so I fixed it by changing the permissions to 740 on those files on the agents and then restarting them. I am now able to see the storage capacity and disk IOPS with no problem.

   Thanks for all your help.

   Posted 3 months ago #
8. 

   thobbs
   Administrator

   Ah, that explains it. There are a couple of ways that the agent could handle that better; I'll open a ticket to look at improving that in the future.

   Thanks for your cooperation and let us know if you have any other problems.

   Posted 3 months ago #

RSS feed for this topic

Reply

You must log in to post.