not resolved
jsevellec
Member
Hi,
OpsCenter is up on my cluster for 2 Days.
It is installed on one of my cassandra node (I knows it not a good idea but it's for test)
It now takes a lot of CPU, memory and cause high load average. My cassandra node (where the opscenter is) crash...
Have you ever seen this behavior?
My OpsCenter is "out of the box". Is there some configurations or tuning to improve it all?
We just released a new version of OpsCenter that fixes some performance issues we found. You can view the release notes here: http://www.datastax.com/docs/opscenter/release_notes#opscenter-1-2-1. Let us know if that doesn't solve your issues.
kirktrue
Member
I'm using Apache's Cassandra 1.0.5 and OpsCenter 1.3.1.
It takes about 1-2 minutes for the UI to start up and we routinely get time out errors in the UI.
Python (running the OpsCenter web application) is constantly sitting at 100% CPU on an 8 core box with 12 GB of RAM.
There's a little connection icon in the top-right corner of the UI and it starts flashing a little red 'X' over the cable.
Any ideas?
thobbs
Administrator
Kirk,
It sounds like something has gone seriously wrong with opscenterd. Can you check the opscenterd log (by default at /var/log/opscenter/opscenterd.log if you installed from a package) to see what's going on?
(The red 'X' indicates that your browser is having trouble communicating with the opscenterd webserver.)
kirktrue
Member
Nothing particularly interesting in the logs apart from the memory usage is growing way too fast:
2012-01-10 09:38:28-0800 [] DEBUG: Average opscenterd CPU usage: 46.38%, max resident: 107 MB
2012-01-10 10:00:28-0800 [] DEBUG: Average opscenterd CPU usage: 99.87%, max resident: 1267 MB
2012-01-10 11:00:29-0800 [] DEBUG: Average opscenterd CPU usage: 99.88%, max resident: 2873 MB
2012-01-10 12:00:34-0800 [] DEBUG: Average opscenterd CPU usage: 99.94%, max resident: 3680 MB
2012-01-10 13:00:29-0800 [] DEBUG: Average opscenterd CPU usage: 99.72%, max resident: 4330 MB
2012-01-10 14:00:33-0800 [] DEBUG: Average opscenterd CPU usage: 99.91%, max resident: 4835 MB
2012-01-10 15:00:29-0800 [] DEBUG: Average opscenterd CPU usage: 99.83%, max resident: 5264 MB
2012-01-10 16:00:28-0800 [] DEBUG: Average opscenterd CPU usage: 99.56%, max resident: 5640 MB
thobbs
Administrator
Having 800 CFs will definitely have a performance impact on OpsCenter and will result in quite a bit of metric data being stored in your cluster. You should probably disable the collection of metrics on those CFs. In opscenterd.conf, this can be done by setting either ignored_keyspaces or ignored_column_familes under [cassandra_metrics].
thobbs
Administrator
I should also add that it's likely that you should think about changing your data model. There are probably other models that would work better than using 800 separate CFs, and would have lower overhead.
You must log in to post.
