Support Forums : DataStax

DataStax Support Forums » OpsCenter

NPE: Error getting devices and partitions

(18 posts) (6 voices)

Started 1 year ago by
Latest reply from kireet
This topic is not resolved

Tags:

12 Next »

Anonymous
Unregistered

On startup of the opscenter agent (1.4), I am seeing the error below in the agent.log file. I'm running cassandra 1.0.7 on a 30 node cluster. Each node host in the cluster is identically configured. Any clues on what may cause this NPE and how to repair it?

I'm not clear if its indicative of larger issue with the cluster.

ERROR [pool-3-thread-3] 2012-02-09 17:35:41,963 Error getting devices and partitions
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1140)
at java.util.regex.Matcher.reset(Matcher.java:291)
at java.util.regex.Matcher.<init>(Matcher.java:211)
at java.util.regex.Pattern.matcher(Pattern.java:888)
at clojure.core$re_matcher.invoke(core.clj:3760)
at clojure.contrib.str_utils2$fn__231.invoke(str_utils2.clj:180)
at clojure.lang.MultiFn.invoke(MultiFn.java:172)
at opsagent.util$escape_device_name.invoke(util.clj:29)
at opsagent.nodedetails$get_disk_info.invoke(nodedetails.clj:102)
at opsagent.nodedetails$run_longtime_collection$fn__1208.invoke(nodedetails.clj:173)
at opsagent.jmx$jmx_wrap.doInvoke(jmx.clj:93)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.nodedetails$run_longtime_collection.invoke(nodedetails.clj:165)
at opsagent.nodedetails$start_pool$fn__1234$fn__1235.invoke(nodedetails.clj:214)
at opsagent.util$error_wrap.doInvoke(util.clj:97)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at opsagent.nodedetails$start_pool$fn__1234.invoke(nodedetails.clj:214)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Posted 1 year ago #
mbulman
Key Master

First, I don't suspect that this is an issue with your cluster itself, so you're okay there. If you could paste the output of the following commands from one of your cassandra nodes, that would be very helpful:

cat /proc/partitions

df <data dir> (<data dir> being the location of data_file_directories configured in cassandra.yaml)

df <commitlog dir> (commitlog dir> being the location of commitlog_directory in cassandra.yaml)

iostat -x

Thanks.

Posted 1 year ago #
Anonymous
Unregistered

OpsCenter Agent is not running as root but runs under same account as CassandraDaemon

/cat/proc/partitions
--------------------

major minor #blocks name

104 0 71652960 cciss/c0d0
104 1 102400 cciss/c0d0p1
104 2 71550496 cciss/c0d0p2
253 0 4194304 dm-0
253 1 4194304 dm-1
253 2 524288 dm-2
253 3 8388608 dm-3
253 4 8224768 dm-4
253 5 2097152 dm-5
253 6 43909120 dm-6

df - Currently same filesystem for commit and data
------------------

Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/rootvg-datavol
ext3 42G 5.4G 34G 14% /abclocal/data

/dev/mapper/rootvg-datavol
ext3 42G 5.4G 34G 14% /abclocal/data

iostat
-----------
avg-cpu: %user %nice %system %iowait %steal %idle
9.00 0.00 0.33 0.06 0.00 90.61

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.03 42.45 0.08 28.05 3.36 527.47 18.87 0.07 2.40 0.26 0.72
cciss/c0d0p1
0.00 0.00 0.00 0.00 0.03 0.00 102.15 0.00 3.34 2.41 0.00
cciss/c0d0p2
0.03 42.45 0.08 28.05 3.32 527.47 18.87 0.07 2.40 0.26 0.72
dm-0 0.00 0.00 0.09 5.10 2.68 40.79 8.38 0.01 2.61 0.13 0.07
dm-1 0.00 0.00 0.01 58.41 0.24 467.25 8.00 0.28 4.72 0.11 0.63
dm-2 0.00 0.00 0.00 6.10 0.00 12.19 2.00 0.01 1.36 0.09 0.06
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 6.58 0.40 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 7.98 0.00 5.34 0.83 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 7.08 0.84 0.00
dm-6 0.00 0.00 0.01 0.90 0.38 7.24 8.34 0.00 2.99 0.22 0.02

Posted 1 year ago #
thobbs
Administrator

Thanks for the output. Would you mind also pasting the results of the following?

* readlink /sys/dev/block/253:0
* readlink /sys/dev/block/104:0
* readlink /sys/dev/block/104:1

Posted 1 year ago #
Anonymous
Unregistered

was this issue resolved - i have the same issue, unfortunately and any help would be appreciated

TIA

Posted 1 year ago #
nickmbailey
Key Master

Naseerm,

This is caused by a bug in the agent code when handling certain device types. Hopefully there will be a fix for this bug in the next release of OpsCenter.

You could help us make sure we fix the bug correctly by giving the output of the following commands on one of your cassandra nodes:

cat /proc/partitions
df <data dir> (<data dir> being the location of data_file_directories configured in cassandra.yaml)
df <commitlog dir> (commitlog dir> being the location of commitlog_directory in cassandra.yaml)
iostat -x
readlink /sys/dev/block/253:0
readlink /sys/dev/block/104:0
readlink /sys/dev/block/104:1

Thanks for the help if you are able,
Nick

Posted 1 year ago #
Anonymous
Unregistered

Hi, apologies for not responding earlier. The results of the readlinks are empty. I'm not sure if this was the results you were expecting.

ls of /sys ->

drwxr-xr-x 3 root root 0 Feb 12 04:07 firmware/
drwxr-xr-x 7 root root 0 Feb 12 04:07 devices/
drwxr-xr-x 2 root root 0 Feb 12 04:08 power/
drwxr-xr-x 35 root root 0 Feb 12 04:08 block/
drwxr-xr-x 4 root root 0 Feb 12 04:08 kernel/
drwxr-xr-x 14 root root 0 Feb 12 04:08 bus/
drwxr-xr-x 30 root root 0 Feb 12 04:09 class/
drwxr-xr-x 3 root root 0 Feb 12 04:09 fs/
drwxr-xr-x 70 root root 0 Feb 12 04:09 module/

ls /sys/block ->

drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-6/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-5/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-4/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-3/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-2/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram9/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram8/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram7/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram6/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram5/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram4/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram3/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram2/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram15/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram14/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram13/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram12/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram11/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram10/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram1/
drwxr-xr-x 4 root root 0 Feb 12 04:08 ram0/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-1/
drwxr-xr-x 4 root root 0 Feb 12 04:08 dm-0/
drwxr-xr-x 7 root root 0 Feb 12 04:08 cciss!c0d0/
drwxr-xr-x 5 root root 0 Feb 12 04:09 md0/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop7/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop6/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop5/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop4/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop3/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop2/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop1/
drwxr-xr-x 4 root root 0 Feb 12 04:09 loop0/

Posted 1 year ago #
thobbs
Administrator

Heskech,

Thanks very much. No, that's not what I was expecting. What Linux distro and release are you running?

Posted 1 year ago #
Anonymous
Unregistered

2.6.18-194.11.4.el5:RHEL5:5.5-1:x86_64:ib-dev

Posted 1 year ago #
Anonymous
Unregistered

thobbs:

[root@lfs03 logs]# cat /proc/partitions

major minor #blocks name

252 0 104857600 vda
252 1 131072 vda1
252 2 1048576 vda2
252 3 1048576 vda3
252 4 102628352 vda4
253 0 2097152 dm-0
253 1 75497472 dm-1
253 2 1048576 dm-2
253 3 2097152 dm-3
[root@lfs03x-dxc1 logs]# iostat -x

Linux 2.6.32-71.el6.x86_64 (lfs03x-dxc1.cimops.net) 02/15/12 _x86_64_ (1 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
10.63 0.00 2.55 2.86 0.00 83.96

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
vda 0.01 2.84 0.05 42.78 1.83 151.65 3.58 0.13 3.09 0.98 4.20
dm-0 0.00 0.00 0.04 0.44 1.23 3.50 9.93 0.23 478.34 0.55 0.03
dm-1 0.00 0.00 0.01 31.40 0.32 142.07 4.53 0.23 7.47 1.26 3.97
dm-2 0.00 0.00 0.00 0.00 0.00 0.02 7.98 0.00 97.35 10.71 0.00
dm-3 0.00 0.00 0.00 0.69 0.02 5.52 8.01 0.01 19.71 3.52 0.24

[root@lfs03x logs]# readlink /sys/dev/block/253:0
../../devices/virtual/block/dm-0
[root@lfs03x logs]# readlink /sys/dev/block/252:0
../../devices/virtio-pci/virtio1/block/vda

thanks

Posted 1 year ago #
thobbs
Administrator

Heskech and Naseerm, thank you both, this has been extremely useful.

Heskech, it's very unusual that you don't have the normal sysfs directory structure. Is this RHEL instance setup in an unusual way? In a particular VM, or as a cloud instance somewhere?

The output of ls /sys/block/cciss!c0d0 would also be useful.

Posted 1 year ago #
thobbs
Administrator

Heskech, the output of rpm -qa | grep sysfs might also be worthwhile.

Posted 1 year ago #
Anonymous
Unregistered

ls -ltr /sys/block/ccis!c0d0

drwxr-xr-x 2 root root 0 Feb 12 04:07 slaves/
drwxr-xr-x 2 root root 0 Feb 12 04:07 holders/
drwxr-xr-x 3 root root 0 Feb 12 04:08 queue/
--w------- 1 root root 4.0K Feb 12 04:08 uevent
lrwxrwxrwx 1 root root 0 Feb 12 04:08 subsystem -> ../../block/
-r--r--r-- 1 root root 4.0K Feb 12 04:08 stat
-r--r--r-- 1 root root 4.0K Feb 12 04:08 size
-r--r--r-- 1 root root 4.0K Feb 12 04:08 removable
-r--r--r-- 1 root root 4.0K Feb 12 04:08 range
-r--r--r-- 1 root root 4.0K Feb 12 04:08 dev
lrwxrwxrwx 1 root root 0 Feb 12 04:08 device -> ../../devices/pci0000:00/0000:00:03.0/0000:0a:00.0/0000:0b:08.0/
drwxr-xr-x 3 root root 0 Feb 12 04:08 cciss!c0d0p2/
drwxr-xr-x 3 root root 0 Feb 12 04:08 cciss!c0d0p1/

rpm -qa | grep sysfs

sysfsutils-2.0.0-6
libsysfs-2.0.0-6
libsysfs-2.0.0-6

Posted 1 year ago #
thobbs
Administrator

Great, that should be enough to fix the issue in an upcoming release. Thanks again.

Posted 1 year ago #
kireet
Member

I am facing this error message as well. Is there a way to confirm that it is the same underlying issue? Is there a workaround until the next release is available or is there a timeline for the next release?

Posted 1 year ago #

RSS feed for this topic

12 Next »

Reply »

You must log in to post.