DataStax Support Forums » Tag: Amazon EC2

DataStax Support Forums » Tag: Amazon EC2 - Recent Posts http://www.datastax.com/support-forums/tags/amazon-ec2 Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 20:51:59 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Problem Disk IO wait." http://www.datastax.com/support-forums/topic/problem-disk-io-wait#post-1877 Wed, 09 May 2012 09:13:42 +0000 Anonymous 1877@http://www.datastax.com/support-forums/ We have the following stats on single node cassandra on Amazon EC2/Rightscale m1.large instance with 2 ephemeral disks with raid0. (7.6 GB Total Memory) 4 GB RAM is allocated to cassandra Heap, 800MB is Heap NEW size. following stats are from OpsCenter community 2.0 Read Requests 285 to 340 per second Write Requests 257 to 720 per second OS Load 15.15 to 17.15 Write Request Latency 293 to 685 micros OS Sent Network Traffic 18 MB to 30 MB per second OS Recieved Network Traffic 22 MB to 34 MB per second OS Disk Queue Size 23 to 26 requests Read Requests Pending 8 to 20 Read Request Latency 69140 to 92885 micros OS Disk latency 37 to 42 ms OS Disk Throughput 12 to 14 Mb per second Disk IOPs Reads 600 to 740 per second Disk IOPs Writes 2 to 7 per second IOWait 60 to 70 % CPU avg Idle 24 to 30 % CPU avg Rowcache is disabled. Are the above stats are satisfying with the provided configuration....OR how could we tweak it more to get less IOWait..........because we think that we are experiencing lots of IOWait.....how could we tweak it to get the best. Read Requests are mixed.........some are from one super column family and one standard having more than million keys......and varying no. of super columns max 14 with varying no. of subcolumns from 1 to 10000 and varying no. of columns max 14 in standard column family...............subcolumns are very thin in nature with 0 bytes value....8 bytes for name. Process is removing the data from super column family and writing the processed data on standard one. Would EBS Disks work better....on Amazon EC2 Anonymous on "Trouble running OpsCenter Agent manually please HELP!!!!" http://www.datastax.com/support-forums/topic/trouble-running-opscenter-agent-manually-please-help-1#post-1808 Tue, 01 May 2012 11:11:34 +0000 Anonymous 1808@http://www.datastax.com/support-forums/ Problem Solved.........i was using an old version of cassandra..........there are some additions in JMX paramters in new version like AllDataLocations etc.........those were missing in my version. Thanks thobbs for your answer...now i have upgraded to latest version cassandra.........and it is working fine.........i read your answer today..........but i had figured it out day before yesterday...........i had to decompile opscenter-agent code...:).....but it was really hard after decompilation......can't we have open source code avaiable on the website for the community edition of opscenter......that will help. thobbs on "Trouble running OpsCenter Agent manually please HELP!!!!" http://www.datastax.com/support-forums/topic/trouble-running-opscenter-agent-manually-please-help-1#post-1801 Mon, 30 Apr 2012 15:06:02 +0000 thobbs 1801@http://www.datastax.com/support-forums/ What version of OpsCenter and Cassandra are you running? If you connect to Cassandra with jconsole and go to MBeans -> org.apache.cassandra.db -> StorageService -> Attributes, do you see fields named "AllDataFileLocations", "CommitLogLocation", and "SavedCachesLocation"? If so, what are their values? (You may need to double click the AllDataFileLocations value.) Anonymous on "Trouble running OpsCenter Agent manually please HELP!!!!" http://www.datastax.com/support-forums/topic/trouble-running-opscenter-agent-manually-please-help-1#post-1793 Sat, 28 Apr 2012 16:48:16 +0000 Anonymous 1793@http://www.datastax.com/support-forums/ I have manually installed agent through Tarball Installations instructions but i am having trouble running agent in foreground as well as background....i am not able to get Disk IO Metrics........everything else is working fine. ERROR [pool-4-thread-3] 2012-04-28 12:15:01,692 Error getting devices and partitions java.io.IOException: Process failed: df Exit val: 1 Output: df: `': No such file or directory at opsagent.proc$handle_proc_results.invoke(proc.clj:68) at opsagent.proc$run_proc.doInvoke(proc.clj:92) at clojure.lang.RestFn.invoke(RestFn.java:410) at opsagent.os.linux_disks$get_partition.invoke(linux_disks.clj:124) at clojure.lang.Var.invoke(Var.java:365) at clojure.lang.AFn.applyToHelper(AFn.java:161) at clojure.lang.Var.applyTo(Var.java:482) at clojure.core$apply.invoke(core.clj:540) at opsagent.os.disks$os_specific.doInvoke(disks.clj:12) at clojure.lang.RestFn.invoke(RestFn.java:423) at opsagent.os.disks$get_partition.invoke(disks.clj:24) at opsagent.nodedetails$get_disk_info.invoke(nodedetails.clj:77) at opsagent.nodedetails$run_longtime_collection$fn__2803.invoke(nodedetails.clj:167) at opsagent.jmx$jmx_wrap.doInvoke(jmx.clj:100) at clojure.lang.RestFn.invoke(RestFn.java:410) at opsagent.nodedetails$run_longtime_collection.invoke(nodedetails.clj:159) at opsagent.nodedetails$start_pool$fn__2829$fn__2830.invoke(nodedetails.clj:210) at opsagent.util$error_wrap.doInvoke(util.clj:114) at clojure.lang.RestFn.invoke(RestFn.java:423) at opsagent.nodedetails$start_pool$fn__2829.invoke(nodedetails.clj:209) at clojure.lang.AFn.run(AFn.java:24) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) ................And................ ERROR [pool-3-thread-7] 2012-04-28 12:16:40,885 Long os-stats collector failed: Process failed: iostat -x -d -m 60 2 Exit val: 130 Output: Linux 2.6.21.7-2.fc8xen (xxxxxxxxxxxxxxx) 04/28/12 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda1 0.13 24.66 0.83 3.80 0.02 0.11 58.17 0.04 8.27 1.26 0.58 sdb 38.54 191.82 307.75 20.05 7.33 0.83 50.96 11.13 33.97 2.53 82.78 sdc 0.00 0.00 0.00 0.00 0.00 0.00 23.52 0.00 1.03 1.00 0.00 ERROR [pool-3-thread-3] 2012-04-28 12:16:40,885 Long os-stats collector failed: Process failed: iostat -c -m 60 2 Exit val: 130 Output: Linux 2.6.21.7-2.fc8xen (xxxxxxxxxxxxxx) 04/28/12 avg-cpu: %user %nice %system %iowait %steal %idle 5.30 0.27 1.74 41.47 1.48 49.74 ********************************************** command df and iostat are working fine when are executed externally. I am running this on Amazon EC2 instance on CENTOS.