DataStax Support Forums » Topic: Failure running Hive examples on DSE

DataStax Support Forums » Topic: Failure running Hive examples on DSE http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse Software, Support, and Training for Apache Cassandra en-US Sat, 18 May 2013 18:41:17 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2061 Fri, 08 Jun 2012 18:15:33 +0000 Anonymous 2061@http://www.datastax.com/support-forums/ When you start up all your nodes, use dsetool movejt to set the jobtracker at the specified datacenter / location. Probably the analytics node is not a seed, so it doesn't start up the jobtracker automatically. It should remember the setting. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2049 Wed, 06 Jun 2012 21:51:57 +0000 Anonymous 2049@http://www.datastax.com/support-forums/ We created a cluster across an ec2 availability zones within a region. We are using EC2Snitch and we are having to bring up the nodes one at a time. What would the proper procedure be fore bringing up a 3 cassandra node, one in each availability zone, then bringing up an analytical node, then possibly adding more nodes as needed. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2047 Wed, 06 Jun 2012 18:05:08 +0000 Anonymous 2047@http://www.datastax.com/support-forums/ How can you tell there is no jobtracker? If I run #dsetool jobtracker 10.10.21.85:8012 I see there is a (TT) in the ring i was wondering if that should be a (JT). If so how do you change that with service des start method. the contents of /etc/default/dse # enable this to start Hadoop's JobTracker and/or TaskTrackers on this machine. # If left disabled, this will act as a regular Cassandra node. HADOOP_ENABLED=1 # enable this to set the replication factor for CFS. Note that this will only # have an effect the first time a cluster is started with HADOOP_ENABLED=1 and # after that will be a no-op. Defaults to 1. #CFS_REPLICATION_FACTOR=1 # enable this to start Solr search indexing on this machine. If left disabled, # this will act as a regular Cassandra node. SOLR_ENABLED=0 ~ Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2046 Wed, 06 Jun 2012 16:25:45 +0000 Anonymous 2046@http://www.datastax.com/support-forums/ What happened to your jobtracker? From dsetool ring you don't seem to be running one. Try starting the analytics node with -j option. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2045 Wed, 06 Jun 2012 13:49:37 +0000 Anonymous 2045@http://www.datastax.com/support-forums/ dsetool ring >>> Address DC Rack Workload Status State Load Owns Token 113427455640312821154458202477256070485 10.10.21.85 us-east 1a Analytics(tt) Up Normal 5.49 MB 33.33% 0 10.120.206.93 us-east 1b Cassandra Up Normal 5.22 MB 33.33% 56713727820156410577229101238628035242 10.68.78.182 us-east 1d Cassandra Up Normal 5.59 MB 33.33% 113427455640312821154458202477256070485 hive.log>> 2012-06-06 01:56:14,558 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(780)) - Got exception: java.io.IOException UnavailableException() 2012-06-06 01:56:14,562 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(783)) - java.io.IOException: UnavailableException() at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:686) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readINodeColumns(CassandraFileSystemThriftStore.java:617) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:569) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:561) at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.getFileStatus(CassandraFileSystem.java:508) at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:377) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1040) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$700(HiveMetaStore.java:143) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1081) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:362) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:400) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:540) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: UnavailableException() at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7764) at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570) at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.datastax.bdp.util.CassandraProxyClient.invoke(CassandraProxyClient.java:407) at $Proxy5.get_slice(Unknown Source) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:668) ... 30 more 2012-06-06 01:56:14,564 ERROR exec.Task (SessionState.java:printError(380)) - FAILED: Error in metadata: MetaException(message:Got exception: java.io.IOException UnavailableException()) org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException UnavailableException()) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:546) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: MetaException(message:Got exception: java.io.IOException UnavailableException()) at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:784) at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:385) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1040) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$700(HiveMetaStore.java:143) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1081) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:362) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:400) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:540) ... 17 more 2012-06-06 01:56:14,565 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2043 Wed, 06 Jun 2012 09:42:26 +0000 Anonymous 2043@http://www.datastax.com/support-forums/ What is your cluster configuration (dsetool ring)? Can you post a full exception stacktrace from hive logs? Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2042 Wed, 06 Jun 2012 02:00:52 +0000 Anonymous 2042@http://www.datastax.com/support-forums/ It seems to be similar to this problem <a href="http://www.datastax.com/support-forums/topic/there-was-a-problem-with-the-cassandra-hive-metastore" rel="nofollow">http://www.datastax.com/support-forums/topic/there-was-a-problem-with-the-cassandra-hive-metastore</a> But the error is slightly different. hive> CREATE TABLE invites (foo INT, bar STRING) > PARTITIONED BY (ds STRING); FAILED: Error in metadata: MetaException(message:Got exception: java.io.IOException UnavailableException()) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask We are trying to run the lastest DSE up on amazon ec2. Any thoughts on our configuration error? Thanks, Todd