Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

Issue with runnig PortfolioDemo App using DSE1.0.1

(16 posts) (4 voices)

Started 1 year ago by
Latest reply from
This topic is resolved

Tags:

No tags yet.

12 Next »

Anonymous
Unregistered

I have downloaded and installed DSE 1.0.1. I am trying to run the PortfolioManager App and following the steps as mentioned in http://www.datastax.com/docs/1.0/datastax_enterprise/portfolio_demo#dse-demo. However, I am unable to see the Step 9 wherein refreshing the Portfolio Manager web page does not show results of Largest Historical 10day Loss for each portfolio.

I am also observing this error in hive.log.

012-02-23 13:22:52,698 ERROR metadata.Hive (Hive.java:getTable(833)) - NoSuchObjectException(message:default.Portfolios table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1040)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1035)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:269)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1035)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:673)

I also do not see anything populated in PortfolioDemo.HistLoss. Other CFs such as Portfolios, Stocks and StockHist has data populated.

Posted 1 year ago #
joaquin
Administrator

Hello Sunit,

Could you ensure that setup was run with sudo throughout? Could you try upgrading to 1.0.2 so that we can get a baseline for the setup?

Thanks,
Joaquin

Posted 1 year ago #
Anonymous
Unregistered

Still waiting to get the credential email for me to download 1.0.2.

Posted 1 year ago #
Anonymous
Unregistered

Joaquin: Yes, the setup was run with sudo

Posted 1 year ago #
mbulman
Key Master

If you still have your credentials from downloading DSE 1.0.1, those will work for DSE 1.0.2. If not, let us know and we'll be sure an email gets sent out to you.

Posted 1 year ago #
Anonymous
Unregistered

I was able to download DSE1.0.2. Thanks :). Then I followed the same steps as mentioned in the site (refer to first message) and still am unable to see the populated values for "Largest Historical 10day Loss: ?" and got this error in hive.log :

2012-03-01 13:35:03,834 ERROR metadata.Hive (Hive.java:getTable(833)) - NoSuchObjectException(message:default.Portfolios table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1040)

Posted 1 year ago #
tjake
Moderator

The 10_day_loss.q file defines that table could you execute that create table statement manually and see what the error is?

Posted 1 year ago #
Anonymous
Unregistered

All the statements passes except statement:"INSERT OVERWRITE TABLE portfolio_returns". Below is the trail:

hive> DROP TABLE IF EXISTS Portfolios;
OK
Time taken: 1.105 seconds
hive> create external table Portfolios(row_key string, column_name string, value string)
> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
> WITH SERDEPROPERTIES ("cassandra.columns.mapping" = ":key,:column,:value",
> "cassandra.ks.name" = "PortfolioDemo",
> "cassandra.ks.repfactor" = "1",
> "cassandra.ks.strategy" = "org.apache.cassandra.locator.SimpleStrategy",
> "cassandra.cf.name" = "Portfolios" ,
> "cassandra.host" = "127.0.0.1" ,
> "cassandra.port" = "9160",
> "cassandra.partitioner" = "org.apache.cassandra.dht.RandomPartitioner")
> TBLPROPERTIES (
> "cassandra.input.split.size" = "64000",
> "cassandra.range.size" = "1000",
> "cassandra.slice.predicate.size" = "1000");
OK
Time taken: 0.302 seconds
hive> DROP TABLE IF EXISTS StockHist;
OK
Time taken: 0.0060 seconds
hive> create external table StockHist(row_key string, column_name string, value double)
> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
> WITH SERDEPROPERTIES ("cassandra.ks.name" = "PortfolioDemo",
> "cassandra.cf.validatorType" = "UTF8Type,UTF8Type,DoubleType"
> );
OK
Time taken: 0.018 seconds
hive> --first calculate returns
> DROP TABLE IF EXISTS 10dayreturns;
OK
Time taken: 0.0060 seconds
hive> CREATE TABLE 10dayreturns(ticker string, rdate string, return double)
> STORED AS SEQUENCEFILE;
OK
Time taken: 0.011 seconds

hive> INSERT OVERWRITE TABLE 10dayreturns
> select a.row_key ticker, b.column_name rdate, (b.value - a.value) ret
> from StockHist a JOIN StockHist b on
> (a.row_key = b.row_key AND date_add(a.column_name,10) = b.column_name);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203021013_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0001
Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0001
2012-03-02 10:25:24,752 Stage-1 map = 0%, reduce = 0%
2012-03-02 10:25:55,284 Stage-1 map = 100%, reduce = 0%
2012-03-02 10:25:56,522 Stage-1 map = 0%, reduce = 0%
2012-03-02 10:26:36,284 Stage-1 map = 0%, reduce = 100%
Ended Job = job_201203021013_0001
Loading data to table default.10dayreturns
Deleted cfs://null/user/hive/warehouse/10dayreturns
Table default.10dayreturns stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 4158622]
90000 Rows loaded to 10dayreturns
OK
Time taken: 93.094 seconds

hive> INSERT OVERWRITE TABLE portfolio_returns
> select row_key portfolio, rdate, SUM(b.return)
> from Portfolios a JOIN 10dayreturns b ON
> (a.column_name = b.ticker)
> group by row_key, rdate;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203021013_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0002
Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0002
2012-03-02 10:27:57,851 Stage-1 map = 0%, reduce = 0%
2012-03-02 10:28:04,941 Stage-1 map = 100%, reduce = 0%
2012-03-02 10:28:06,191 Stage-1 map = 0%, reduce = 0%
2012-03-02 10:28:14,677 Stage-1 map = 100%, reduce = 0%
2012-03-02 10:28:22,915 Stage-1 map = 100%, reduce = 17%
2012-03-02 10:28:29,138 Stage-1 map = 100%, reduce = 67%
2012-03-02 10:28:32,288 Stage-1 map = 100%, reduce = 68%
2012-03-02 10:28:38,316 Stage-1 map = 100%, reduce = 73%
2012-03-02 10:28:41,468 Stage-1 map = 100%, reduce = 80%
2012-03-02 10:28:46,135 Stage-1 map = 100%, reduce = 90%
2012-03-02 10:28:47,140 Stage-1 map = 100%, reduce = 95%
2012-03-02 10:28:48,146 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201203021013_0002
Launching Job 2 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203021013_0003, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0003
Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0003
2012-03-02 10:28:57,248 Stage-2 map = 0%, reduce = 0%
2012-03-02 10:29:17,341 Stage-2 map = 100%, reduce = 100%
Ended Job = job_201203021013_0003 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive> DROP TABLE IF EXISTS HistLoss;
OK
Time taken: 0.0060 seconds
hive> create external table HistLoss(row_key string, worst_date string, loss string)
> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
> WITH SERDEPROPERTIES ("cassandra.ks.name" = "PortfolioDemo");
OK
Time taken: 0.08 seconds
hive> INSERT OVERWRITE TABLE HistLoss
> select a.portfolio, rdate, cast(minp as STRING)
> FROM (
> select portfolio, MIN(preturn) as minp
> FROM portfolio_returns
> group by portfolio
> ) a JOIN portfolio_returns b ON (a.portfolio = b.portfolio and a.minp = b.preturn);
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203021013_0004, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0004
Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0004
2012-03-02 10:34:50,648 Stage-1 map = 0%, reduce = 0%
2012-03-02 10:34:51,655 Stage-1 map = 100%, reduce = 0%
2012-03-02 10:34:59,686 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201203021013_0004
Launching Job 2 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203021013_0005, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203021013_0005
Kill Command = /Users/srandhawa/datastax/dse-1.0.2/bin/dse hadoop job -Dmapred.job.tracker=127.0.0.1:8012 -kill job_201203021013_0005
2012-03-02 10:35:05,026 Stage-0 map = 0%, reduce = 0%
2012-03-02 10:35:07,039 Stage-0 map = 50%, reduce = 0%
2012-03-02 10:35:12,057 Stage-0 map = 100%, reduce = 0%
2012-03-02 10:35:20,090 Stage-0 map = 100%, reduce = 17%
2012-03-02 10:35:21,111 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201203021013_0005
OK
Time taken: 38.27 seconds

Posted 1 year ago #
Anonymous
Unregistered

Also, overwrite tables 10dayreturns and portfolio_returns are not part of PortfolioDemo KS?

Posted 1 year ago #
tjake
Moderator

portfolio_returns is a hive table, not accessible from cassandra. Could you look at the jobtracker and see what the error was in that job?

Posted 1 year ago #
Anonymous
Unregistered

JobTracker is showing this message: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201203021013_0003_m_000001

Is it because of this HistLoss CF in Cassandra is not getting populated which is why I am not seeing anything populated for "Largest Historical 10day Loss" in the chart?

Posted 1 year ago #
tjake
Moderator

That is saying it failed, but to find out why you would need to go to the failed task in the JobTracker and look at the error log screen.

Posted 1 year ago #
Anonymous
Unregistered

I am browsing to this page at http://localhost:50060/logs/userlogs/ and then going to job_201203051010_0002 (the failed job id) and clicking attempt_201203051010_0002_m_000002_0/ wherein I see this message:

HTTP ERROR 404

Problem accessing /logs/userlogs/job_201203051010_0002/attempt_201203051010_0002_m_000002_0/. Reason:

NOT_FOUND

Please let me know if I am looking at right location for error log screen. Please Let me know if you'd like me to share the screen shots.

Thanks for all your help!

Posted 1 year ago #
tjake
Moderator

Based on the error the job that failed is: job_201203021013_0003

If you go to the Jobtracker UI :50030 and check this job's failed mappers the logs will be linked there with the full stacktrace. I apologize for the delay I'm not getting emailed when you reply.

You can also zip up the entire hadoop log dir and send me that if you want :)

My email is jake [at] datastax

Posted 1 year ago #
Anonymous
Unregistered

Thanks Jake. Will send the logs data.

Posted 1 year ago #

RSS feed for this topic

12 Next »

Reply »

You must log in to post.