http://www.datastax.com/support-forums/topic/disk-iops-no-data Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 19:13:36 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-7045 Mon, 22 Oct 2012 08:53:18 +0000 Maffen 7045@http://www.datastax.com/support-forums/ <p>Hi Tupton,</p> <p>Everything works OK now. Thanks for your help.</p> <p>Kind regards,<br /> Maffen </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6975 Tue, 16 Oct 2012 15:15:15 +0000 tupton 6975@http://www.datastax.com/support-forums/ <p>Maffen,</p> <p>The log4j warnings are a known issue at this time, but logging should work regardless. Are you able to see Disk IOPs and storage capacity after getting the right logging config and fixing the commitlog directory permissions, or do you still have an issue? </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6959 Mon, 15 Oct 2012 17:34:12 +0000 Maffen 6959@http://www.datastax.com/support-forums/ <p>Hi,</p> <p>I did a tarball installation<br /> I'm having strange problems with the logging.</p> <p>If I restart the opscenter agent I get the following:<br /> <pre><code>[root@bemobile-cass0 ~]# service opscenter-agent restart Restarting Cassandra cluster manager agent opscenter-agent Stopping opscenter-agent-monitor [ OK ] Stopping opscenter-agent [ OK ] Starting opscenter-agent [ OK ] [root@bemobile-cass0 ~]# log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log). log4j:WARN Please initialize the log4j system properly. log4j:WARN See <a href="http://logging.apache.org/log4j/1.2/faq.html#noconfig" rel="nofollow">http://logging.apache.org/log4j/1.2/faq.html#noconfig</a> for more info.</code></pre> <p>This is with the default installation on all the nodes...</p> <p>Also I just noticed there are 2 (different) log4j.properties.<br /> - /usr/share/opscenter-agent/conf<br /> - /etc/opscenter-agent/</p> <p>But these are 2 different configurations. It seems the init-script is using the one in /usr/share/opscenter-agent/conf.<br /> But there are no log4j messages:<br /> <pre><code>[root@bemobile-cass0 conf]# cat /var/log/opscenter-agent/agent.log nohup: ignoring input Starting OpsCenter agent monitor opscenter_agent_monitor nohup: ignoring input Starting OpsCenter agent monitor opscenter_agent_monitor nohup: ignoring input Starting OpsCenter agent monitor opscenter_agent_monitor</code></pre> <p>Now I soft-linked to the one in /etc/opscenter-agent/ and I am now seeing log messages in /var/log/opscenter-agent/agent.log. :) (However, i am still getting the same WARN messages from log4j)</p> <p>Now with the log messages activated I saw a problem with permissions about the commitlog directory, which are fixed now. And it works now :)</p> <p>Conclusion: The only problem I really had is the default log4j configuration not being valid. Is this a known problem with opscenter tarball installation on CentOS6?</p> <p>Edit: I also have to mention i upgraded from 1.4.1 to 2.1.2, was there a change of log4j configuration between these versions? </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6957 Mon, 15 Oct 2012 15:52:08 +0000 thobbs 6957@http://www.datastax.com/support-forums/ <p>@Maffen</p> <p>Can you attach the logs for one of the agents? If you installed the agents through the OpsCenter interface, they should be located at <code>/var/log/opscenter-agent/agent.log</code>. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6955 Mon, 15 Oct 2012 14:57:17 +0000 Maffen 6955@http://www.datastax.com/support-forums/ <p>Hi,</p> <p>I'm having the same problem on a 6-node CentOS 6.2 cluster.<br /> The opscenterd daemon is running on a seperate virtual machine who is running CentOS 6.3.</p> <p>I've already tried several times restarting all agents and opscenterd, but no luck...</p> <p>Only Disk IOPS and Storage capacity on the dashboard are not working, other metrics are showing fine.</p> <p>Another (maybe related problem) I have is when i restart a casssandra daemon on a node, I also have to restart the opscenter-agent, otherwise the stats are frozen for that node. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6954 Mon, 15 Oct 2012 14:24:52 +0000 schalla 6954@http://www.datastax.com/support-forums/ <p>I highly doubt that. The cluster has been up for several days along with the OpsCenter and its agents. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6918 Fri, 12 Oct 2012 22:18:40 +0000 thobbs 6918@http://www.datastax.com/support-forums/ <p>Okay, that's interesting. When you first start OpsCenter and the agents, it usually takes two or three minutes for disk metrics to show up. Perhaps that's what you were seeing? </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6917 Fri, 12 Oct 2012 21:55:21 +0000 schalla 6917@http://www.datastax.com/support-forums/ <p>It is working fine after restarting OpsCenter daemon and all the agents. I still do not know the root cause yet. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6913 Fri, 12 Oct 2012 19:20:40 +0000 thobbs 6913@http://www.datastax.com/support-forums/ <p>Hmmm, based on what you've posted so far, it seems like things should be working. Can you curl '<code>&lt;opscenter_ip&gt;:8888/&lt;cluster_name&gt;/metrics/&lt;ip&gt;/os-disk-utilization/all?start=1350068040&amp;end=1350068200&amp;step=1</code>' and paste the results here? Replace <code>&lt;ip&gt;</code> with the IP address of one of your two nodes. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6911 Fri, 12 Oct 2012 18:23:49 +0000 schalla 6911@http://www.datastax.com/support-forums/ <p>Here it is. </p> <p> 8 0 262144000 sda<br /> 8 1 200781 sda1<br /> 8 2 157083570 sda2<br /> 8 3 104856255 sda3<br /> 253 0 6291456 dm-0<br /> 253 1 104857600 dm-1<br /> 253 2 2097152 dm-2<br /> 253 3 115343360 dm-3<br /> 253 4 10485760 dm-4<br /> 253 5 4194304 dm-5 </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6908 Fri, 12 Oct 2012 15:32:14 +0000 thobbs 6908@http://www.datastax.com/support-forums/ <p>Thanks for the info; I have an idea of what might be going wrong here. Can I also get the output of '<code>cat /proc/partitions</code>'? </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6903 Fri, 12 Oct 2012 14:36:00 +0000 schalla 6903@http://www.datastax.com/support-forums/ <p>Any ideas/suggestions are highly appreciated. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6870 Thu, 11 Oct 2012 15:12:32 +0000 schalla 6870@http://www.datastax.com/support-forums/ <p>I have put the output on pastebin</p> <p>iostat -x<br /> <a href="http://pastebin.com/Fv7AThGH" rel="nofollow">http://pastebin.com/Fv7AThGH</a></p> <p>&lt;opscenter_ip&gt;:8888/&lt;cluster_name&gt;/nodes?<br /> <a href="http://pastebin.com/vhGBRhYi" rel="nofollow">http://pastebin.com/vhGBRhYi</a> </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6869 Thu, 11 Oct 2012 14:54:13 +0000 thobbs 6869@http://www.datastax.com/support-forums/ <p>Can I get the output of '<code>iostat -x</code>' on one of your Cassandra nodes? Can you also paste the result of <code>curl &lt;opscenter_ip&gt;:8888/&lt;cluster_name&gt;/nodes</code>? Replace <code>&lt;cluster_name&gt;</code> with the name of the cluster, with whitespace and special characters replaced by underscores. </p> http://www.datastax.com/support-forums/topic/disk-iops-no-data#post-6852 Wed, 10 Oct 2012 21:17:10 +0000 schalla 6852@http://www.datastax.com/support-forums/ <p>I do not see any data on the main dashboard on OpsCenter 2.1. I see other performance metrics like Cluster Reads &amp; Writes, Cluster Latency etc. Is this a configuration issue?<br /> I am running on RHEL 64bit 5.8 </p>