Disk IOPS - NO DATA

(15 posts) (4 voices)

• Started 7 months ago by schalla
• Latest reply from Maffen
• This topic is resolved

Tags:

No tags yet.

1. 

   schalla
   Member

   I do not see any data on the main dashboard on OpsCenter 2.1. I see other performance metrics like Cluster Reads & Writes, Cluster Latency etc. Is this a configuration issue?
   I am running on RHEL 64bit 5.8

   Posted 7 months ago #
2. 

   thobbs
   Administrator

   Can I get the output of 'iostat -x' on one of your Cassandra nodes? Can you also paste the result of curl <opscenter_ip>:8888/<cluster_name>/nodes? Replace <cluster_name> with the name of the cluster, with whitespace and special characters replaced by underscores.

   Posted 7 months ago #
3. 

   schalla
   Member

   I have put the output on pastebin

   iostat -x
   http://pastebin.com/Fv7AThGH

   <opscenter_ip>:8888/<cluster_name>/nodes?
   http://pastebin.com/vhGBRhYi

   Posted 7 months ago #
4. 

   schalla
   Member

   Any ideas/suggestions are highly appreciated.

   Posted 7 months ago #
5. 

   thobbs
   Administrator

   Thanks for the info; I have an idea of what might be going wrong here. Can I also get the output of 'cat /proc/partitions'?

   Posted 7 months ago #
6. 

   schalla
   Member

   Here it is.

   8 0 262144000 sda
   8 1 200781 sda1
   8 2 157083570 sda2
   8 3 104856255 sda3
   253 0 6291456 dm-0
   253 1 104857600 dm-1
   253 2 2097152 dm-2
   253 3 115343360 dm-3
   253 4 10485760 dm-4
   253 5 4194304 dm-5

   Posted 7 months ago #
7. 

   thobbs
   Administrator

   Hmmm, based on what you've posted so far, it seems like things should be working. Can you curl '<opscenter_ip>:8888/<cluster_name>/metrics/<ip>/os-disk-utilization/all?start=1350068040&end=1350068200&step=1' and paste the results here? Replace <ip> with the IP address of one of your two nodes.

   Posted 7 months ago #
8. 

   schalla
   Member

   It is working fine after restarting OpsCenter daemon and all the agents. I still do not know the root cause yet.

   Posted 7 months ago #
9. 

   thobbs
   Administrator

   Okay, that's interesting. When you first start OpsCenter and the agents, it usually takes two or three minutes for disk metrics to show up. Perhaps that's what you were seeing?

   Posted 7 months ago #
10. 

    schalla
    Member

    I highly doubt that. The cluster has been up for several days along with the OpsCenter and its agents.

    Posted 7 months ago #
11. 

    Maffen
    Member

    Hi,

    I'm having the same problem on a 6-node CentOS 6.2 cluster.
    The opscenterd daemon is running on a seperate virtual machine who is running CentOS 6.3.

    I've already tried several times restarting all agents and opscenterd, but no luck...

    Only Disk IOPS and Storage capacity on the dashboard are not working, other metrics are showing fine.

    Another (maybe related problem) I have is when i restart a casssandra daemon on a node, I also have to restart the opscenter-agent, otherwise the stats are frozen for that node.

    Posted 7 months ago #
12. 

    thobbs
    Administrator

    @Maffen

    Can you attach the logs for one of the agents? If you installed the agents through the OpsCenter interface, they should be located at /var/log/opscenter-agent/agent.log.

    Posted 7 months ago #
13. 

    Maffen
    Member

    Hi,

    I did a tarball installation
    I'm having strange problems with the logging.

    If I restart the opscenter agent I get the following:
    

    [root@bemobile-cass0 ~]# service opscenter-agent restart
    Restarting Cassandra cluster manager agent opscenter-agent
    Stopping opscenter-agent-monitor                           [  OK  ]
    Stopping opscenter-agent                                   [  OK  ]
    Starting opscenter-agent                                   [  OK  ]
    [root@bemobile-cass0 ~]# log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

    This is with the default installation on all the nodes...

    Also I just noticed there are 2 (different) log4j.properties.
    - /usr/share/opscenter-agent/conf
    - /etc/opscenter-agent/

    But these are 2 different configurations. It seems the init-script is using the one in /usr/share/opscenter-agent/conf.
    But there are no log4j messages:
    

    [root@bemobile-cass0 conf]# cat /var/log/opscenter-agent/agent.log
    nohup: ignoring input
    Starting OpsCenter agent monitor opscenter_agent_monitor
    nohup: ignoring input
    Starting OpsCenter agent monitor opscenter_agent_monitor
    nohup: ignoring input
    Starting OpsCenter agent monitor opscenter_agent_monitor

    Now I soft-linked to the one in /etc/opscenter-agent/ and I am now seeing log messages in /var/log/opscenter-agent/agent.log. :) (However, i am still getting the same WARN messages from log4j)

    Now with the log messages activated I saw a problem with permissions about the commitlog directory, which are fixed now. And it works now :)

    Conclusion: The only problem I really had is the default log4j configuration not being valid. Is this a known problem with opscenter tarball installation on CentOS6?

    Edit: I also have to mention i upgraded from 1.4.1 to 2.1.2, was there a change of log4j configuration between these versions?

    Posted 7 months ago #
14. 

    tupton
    Moderator

    Maffen,

    The log4j warnings are a known issue at this time, but logging should work regardless. Are you able to see Disk IOPs and storage capacity after getting the right logging config and fixing the commitlog directory permissions, or do you still have an issue?

    Posted 7 months ago #
15. 

    Maffen
    Member

    Hi Tupton,

    Everything works OK now. Thanks for your help.

    Kind regards,
    Maffen

    Posted 7 months ago #

RSS feed for this topic

Reply

You must log in to post.