DataStax Support Forums » Topic: Issue with runnig PortfolioDemo App using DSE1.0.1

DataStax Support Forums » Topic: Issue with runnig PortfolioDemo App using DSE1.0.1 http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101 Software, Support, and Training for Apache Cassandra en-US Sun, 19 May 2013 07:12:32 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101/page/2#post-1362 Tue, 13 Mar 2012 19:06:16 +0000 Anonymous 1362@http://www.datastax.com/support-forums/ Thanks Jake for your help. The issue was fixed by removing "~" from the cassandra.yaml since it accepts the absolute directory path for the data, commitlogs directories. The errors I was getting was obfuscated because it could not find such data directories and was not throwing such error. Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1348 Fri, 09 Mar 2012 18:59:39 +0000 Anonymous 1348@http://www.datastax.com/support-forums/ Thanks Jake. Will send the logs data. tjake on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1340 Fri, 09 Mar 2012 02:59:32 +0000 tjake 1340@http://www.datastax.com/support-forums/ Based on the error the job that failed is: job_201203021013_0003 If you go to the Jobtracker UI :50030 and check this job's failed mappers the logs will be linked there with the full stacktrace. I apologize for the delay I'm not getting emailed when you reply. You can also zip up the entire hadoop log dir and send me that if you want :) My email is jake [at] datastax Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1289 Mon, 05 Mar 2012 18:35:47 +0000 Anonymous 1289@http://www.datastax.com/support-forums/ I am browsing to this page at <a href="http://localhost:50060/logs/userlogs/" rel="nofollow">http://localhost:50060/logs/userlogs/</a> and then going to job_201203051010_0002 (the failed job id) and clicking attempt_201203051010_0002_m_000002_0/ wherein I see this message: HTTP ERROR 404 Problem accessing /logs/userlogs/job_201203051010_0002/attempt_201203051010_0002_m_000002_0/. Reason: NOT_FOUND Please let me know if I am looking at right location for error log screen. Please Let me know if you'd like me to share the screen shots. Thanks for all your help! tjake on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1283 Sat, 03 Mar 2012 17:37:31 +0000 tjake 1283@http://www.datastax.com/support-forums/ That is saying it failed, but to find out why you would need to go to the failed task in the JobTracker and look at the error log screen. Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1282 Sat, 03 Mar 2012 01:19:57 +0000 Anonymous 1282@http://www.datastax.com/support-forums/ JobTracker is showing this message: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201203021013_0003_m_000001 Is it because of this HistLoss CF in Cassandra is not getting populated which is why I am not seeing anything populated for "Largest Historical 10day Loss" in the chart? tjake on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1281 Sat, 03 Mar 2012 01:09:49 +0000 tjake 1281@http://www.datastax.com/support-forums/ portfolio_returns is a hive table, not accessible from cassandra. Could you look at the jobtracker and see what the error was in that job? Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1280 Fri, 02 Mar 2012 21:55:30 +0000 Anonymous 1280@http://www.datastax.com/support-forums/ Also, overwrite tables 10dayreturns and portfolio_returns are not part of PortfolioDemo KS? Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1278 Fri, 02 Mar 2012 18:41:32 +0000 Anonymous 1278@http://www.datastax.com/support-forums/ All the statements passes except statement:"INSERT OVERWRITE TABLE portfolio_returns". Below is the trail: hive> DROP TABLE IF EXISTS Portfolios; OK Time taken: 1.105 seconds hive> create external table Portfolios(row_key string, column_name string, value string) > STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' > WITH SERDEPROPERTIES ("cassandra.columns.mapping" = ":key,:column,:value", > "cassandra.ks.name" = "PortfolioDemo", > "cassandra.ks.repfactor" = "1", > "cassandra.ks.strategy" = "org.apache.cassandra.locator.SimpleStrategy", > "cassandra.cf.name" = "Portfolios" , > "cassandra.host" = "127.0.0.1" , > "cassandra.port" = "9160", > "cassandra.partitioner" = "org.apache.cassandra.dht.RandomPartitioner") > TBLPROPERTIES ( > "cassandra.input.split.size" = "64000", > "cassandra.range.size" = "1000", > "cassandra.slice.predicate.size" = "1000"); OK Time taken: 0.302 seconds hive> DROP TABLE IF EXISTS StockHist; OK Time taken: 0.0060 seconds hive> create external table StockHist(row_key string, column_name string, value double) > STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' > WITH SERDEPROPERTIES ("cassandra.ks.name" = "PortfolioDemo", > "cassandra.cf.validatorType" = "UTF8Type,UTF8Type,DoubleType" > ); OK Time taken: 0.018 seconds hive> --first calculate returns > DROP TABLE IF EXISTS 10dayreturns; OK Time taken: 0.0060 seconds hive> CREATE TABLE 10dayreturns(ticker string, rdate string, return double) > STORED AS SEQUENCEFILE; OK Time taken: 0.011 seconds hive> INSERT OVERWRITE TABLE 10dayreturns > select a.row_key ticker, b.column_name rdate, (b.value - a.value) ret > from StockHist a JOIN StockHist b on > (a.row_key = b.row_key AND date_add(a.column_name,10) = b.column_name); Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203021013_0001, Tracking URL = <a href="http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0001" rel="nofollow">http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0001</a> Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0001 2012-03-02 10:25:24,752 Stage-1 map = 0%, reduce = 0% 2012-03-02 10:25:55,284 Stage-1 map = 100%, reduce = 0% 2012-03-02 10:25:56,522 Stage-1 map = 0%, reduce = 0% 2012-03-02 10:26:36,284 Stage-1 map = 0%, reduce = 100% Ended Job = job_201203021013_0001 Loading data to table default.10dayreturns Deleted cfs://null/user/hive/warehouse/10dayreturns Table default.10dayreturns stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 4158622] 90000 Rows loaded to 10dayreturns OK Time taken: 93.094 seconds hive> INSERT OVERWRITE TABLE portfolio_returns > select row_key portfolio, rdate, SUM(b.return) > from Portfolios a JOIN 10dayreturns b ON > (a.column_name = b.ticker) > group by row_key, rdate; Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203021013_0002, Tracking URL = <a href="http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0002" rel="nofollow">http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0002</a> Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0002 2012-03-02 10:27:57,851 Stage-1 map = 0%, reduce = 0% 2012-03-02 10:28:04,941 Stage-1 map = 100%, reduce = 0% 2012-03-02 10:28:06,191 Stage-1 map = 0%, reduce = 0% 2012-03-02 10:28:14,677 Stage-1 map = 100%, reduce = 0% 2012-03-02 10:28:22,915 Stage-1 map = 100%, reduce = 17% 2012-03-02 10:28:29,138 Stage-1 map = 100%, reduce = 67% 2012-03-02 10:28:32,288 Stage-1 map = 100%, reduce = 68% 2012-03-02 10:28:38,316 Stage-1 map = 100%, reduce = 73% 2012-03-02 10:28:41,468 Stage-1 map = 100%, reduce = 80% 2012-03-02 10:28:46,135 Stage-1 map = 100%, reduce = 90% 2012-03-02 10:28:47,140 Stage-1 map = 100%, reduce = 95% 2012-03-02 10:28:48,146 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203021013_0002 Launching Job 2 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203021013_0003, Tracking URL = <a href="http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0003" rel="nofollow">http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0003</a> Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0003 2012-03-02 10:28:57,248 Stage-2 map = 0%, reduce = 0% 2012-03-02 10:29:17,341 Stage-2 map = 100%, reduce = 100% Ended Job = job_201203021013_0003 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> DROP TABLE IF EXISTS HistLoss; OK Time taken: 0.0060 seconds hive> create external table HistLoss(row_key string, worst_date string, loss string) > STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' > WITH SERDEPROPERTIES ("cassandra.ks.name" = "PortfolioDemo"); OK Time taken: 0.08 seconds hive> INSERT OVERWRITE TABLE HistLoss > select a.portfolio, rdate, cast(minp as STRING) > FROM ( > select portfolio, MIN(preturn) as minp > FROM portfolio_returns > group by portfolio > ) a JOIN portfolio_returns b ON (a.portfolio = b.portfolio and a.minp = b.preturn); Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203021013_0004, Tracking URL = <a href="http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0004" rel="nofollow">http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0004</a> Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0004 2012-03-02 10:34:50,648 Stage-1 map = 0%, reduce = 0% 2012-03-02 10:34:51,655 Stage-1 map = 100%, reduce = 0% 2012-03-02 10:34:59,686 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203021013_0004 Launching Job 2 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203021013_0005, Tracking URL = <a href="http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0005" rel="nofollow">http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0005</a> Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0005 2012-03-02 10:35:05,026 Stage-0 map = 0%, reduce = 0% 2012-03-02 10:35:07,039 Stage-0 map = 50%, reduce = 0% 2012-03-02 10:35:12,057 Stage-0 map = 100%, reduce = 0% 2012-03-02 10:35:20,090 Stage-0 map = 100%, reduce = 17% 2012-03-02 10:35:21,111 Stage-0 map = 100%, reduce = 100% Ended Job = job_201203021013_0005 OK Time taken: 38.27 seconds tjake on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1275 Fri, 02 Mar 2012 14:55:56 +0000 tjake 1275@http://www.datastax.com/support-forums/ The 10_day_loss.q file defines that table could you execute that create table statement manually and see what the error is? Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1267 Thu, 01 Mar 2012 21:49:10 +0000 Anonymous 1267@http://www.datastax.com/support-forums/ I was able to download DSE1.0.2. Thanks :). Then I followed the same steps as mentioned in the site (refer to first message) and still am unable to see the populated values for "Largest Historical 10day Loss: ?" and got this error in hive.log : 2012-03-01 13:35:03,834 ERROR metadata.Hive (Hive.java:getTable(833)) - NoSuchObjectException(message:default.Portfolios table not found) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1040) mbulman on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1266 Thu, 01 Mar 2012 15:24:46 +0000 mbulman 1266@http://www.datastax.com/support-forums/ If you still have your credentials from downloading DSE 1.0.1, those will work for DSE 1.0.2. If not, let us know and we'll be sure an email gets sent out to you. Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1265 Thu, 01 Mar 2012 01:35:28 +0000 Anonymous 1265@http://www.datastax.com/support-forums/ Joaquin: Yes, the setup was run with sudo Anonymous on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1263 Thu, 01 Mar 2012 00:19:14 +0000 Anonymous 1263@http://www.datastax.com/support-forums/ Still waiting to get the credential email for me to download 1.0.2. joaquin on "Issue with runnig PortfolioDemo App using DSE1.0.1" http://www.datastax.com/support-forums/topic/issue-with-runnig-portfoliodemo-app-using-dse101#post-1261 Wed, 29 Feb 2012 23:29:42 +0000 joaquin 1261@http://www.datastax.com/support-forums/ Hello Sunit, Could you ensure that setup was run with sudo throughout? Could you try upgrading to 1.0.2 so that we can get a baseline for the setup? Thanks, Joaquin