DataStax Support Forums » Tag: hive

DataStax Support Forums » Tag: hive - Recent Posts http://www.datastax.com/support-forums/tags/hive Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 13:34:37 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php dipesh on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table/page/2#post-9072 Tue, 19 Feb 2013 17:02:21 +0000 dipesh 9072@http://www.datastax.com/support-forums/ Finally, I am back to this. Sorry for the delay. Composite Column Name = UTF8Type + LongType/IntegerType/UTF8Type Note: For all above combinations of second column, I am getting the same exception. My Doubts: - Looking at the exception log, "value" seems pretty odd. Is that a problem? - createCompositeKey method in the UDF java code, value of third (end-of-component) and fourth (lastIsOne) is hard coded to '0' and 'true' respectively. Frankly, I didn't understand it completely, even when I read the java-doc of Composite type several times. Please advise. Here is the Exception from the Log: <pre><code>2013-02-19 22:15:32,802 null map = 100%, reduce = 0% [2013-02-19 22:15:34,916] FATAL {ExecReducer} - org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"CSCO","_col1":"price","_col2":1360821839400},"value":{"_col0":{"count":1,"sum":25.13}},"alias":0} at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: InvalidRequestException(why:Not enough bytes to read value of component 0) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:603) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:959) at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:798) at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:724) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247) ... 3 more Caused by: java.io.IOException: InvalidRequestException(why:Not enough bytes to read value of component 0) at org.apache.hadoop.hive.cassandra.output.CassandraAbstractPut.commitChanges(CassandraAbstractPut.java:69) at org.apache.hadoop.hive.cassandra.output.CassandraPut.write(CassandraPut.java:139) at org.apache.hadoop.hive.cassandra.output.HiveCassandraOutputFormat$1.write(HiveCassandraOutputFormat.java:69) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:589) ... 16 more</code></pre> Here is my UDF based on your sample code. <pre><code>package com.cisco.iep.hive.plugins; import java.nio.ByteBuffer; import org.apache.cassandra.utils.ByteBufferUtil; import org.apache.hadoop.hive.ql.exec.UDF; import org.apache.hadoop.io.BytesWritable; public class WriteCompositeStringLong extends UDF{ public BytesWritable evaluate(final String strCol, final Long longCol) { ByteBuffer byteBuff = createCompositeKey(strCol, longCol, 0, false); return new BytesWritable(byteBuff.array()); } private ByteBuffer createCompositeKey(String strCol, Long longCol, int endOfComponent, boolean lastIsOne) { ByteBuffer bytes = ByteBufferUtil.bytes(strCol); int totalSize = 0; if (strCol != null) { totalSize += 2 + bytes.remaining() + 1; if (longCol != null) { totalSize += 2 + 8 + 1; if (endOfComponent != -1) { totalSize += 2 + 1 + 1; } } } ByteBuffer bb = ByteBuffer.allocate(totalSize); if (strCol != null) { bb.putShort((short) bytes.remaining()); bb.put(bytes); bb.put(longCol == null && lastIsOne ? (byte) 1 : (byte) 0); if (longCol != null) { //bb.putShort((short) 16); //bb.put(UUIDGen.decompose(intCol)); bb.putShort((short) 8); //8 for Long value bb.putLong(longCol); bb.put(endOfComponent == -1 && lastIsOne ? (byte) 1 : (byte) 0); if (endOfComponent != -1) { // We are putting a byte only because our test use ints that // fit in a byte *and* IntegerType.fromString() will // return something compatible (i.e, putting a full int here // would break 'fromStringTest') bb.putShort((short) 1); bb.put((byte) endOfComponent); bb.put(lastIsOne ? (byte) 1 : (byte) 0); } } } bb.rewind(); return bb; } }</code></pre> I could paste the all other missing pieces like Hive Create Table script, Source Table details etc. I feel guilty of not resolving this issue yet even after your so much help from you already. Really appreciate your time and help over this!! dipesh on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table/page/2#post-8989 Mon, 11 Feb 2013 05:23:03 +0000 dipesh 8989@http://www.datastax.com/support-forums/ Thanks again, however I will need couple of more days to try out due to other priorities. alexliu on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table#post-8945 Wed, 06 Feb 2013 17:56:32 +0000 alexliu 8945@http://www.datastax.com/support-forums/ Here is composite column encoding according to the source code /* * The encoding of a CompositeType column name should be: * <component><component><component> ... * where <component> is: * <length of value><value><'end-of-component' byte> * where <length of value> is a 2 bytes unsigned short the and the * 'end-of-component' byte should always be 0 for actual column name. * However, it can set to 1 for query bounds. This allows to query for the * equivalent of 'give me the full super-column'. That is, if during a slice * query uses: * start = <3><"foo".getBytes()><0> * end = <3><"foo".getBytes()><1> * then he will be sure to get *all* the columns whose first component is "foo". * If for a component, the 'end-of-component' is != 0, there should not be any * following component. The end-of-component can also be -1 to allow * non-inclusive query. For instance: * start = <3><"foo".getBytes()><-1> * allows to query everything that is greater than <3><"foo".getBytes()>, but * not <3><"foo".getBytes()> itself. */ here is a sample code to create encoding composite column private ByteBuffer createCompositeKey(String s, UUID uuid, int i, boolean lastIsOne) { ByteBuffer bytes = ByteBufferUtil.bytes(s); int totalSize = 0; if (s != null) { totalSize += 2 + bytes.remaining() + 1; if (uuid != null) { totalSize += 2 + 16 + 1; if (i != -1) { totalSize += 2 + 1 + 1; } } } ByteBuffer bb = ByteBuffer.allocate(totalSize); if (s != null) { bb.putShort((short) bytes.remaining()); bb.put(bytes); bb.put(uuid == null && lastIsOne ? (byte)1 : (byte)0); if (uuid != null) { bb.putShort((short) 16); bb.put(UUIDGen.decompose(uuid)); bb.put(i == -1 && lastIsOne ? (byte)1 : (byte)0); if (i != -1) { // We are putting a byte only because our test use ints that fit in a byte *and* IntegerType.fromString() will // return something compatible (i.e, putting a full int here would break 'fromStringTest') bb.putShort((short) 1); bb.put((byte)i); bb.put(lastIsOne ? (byte)1 : (byte)0); } } } bb.rewind(); return bb; } You need change your code to the same encoding as the composite column name dipesh on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table#post-8942 Wed, 06 Feb 2013 16:58:47 +0000 dipesh 8942@http://www.datastax.com/support-forums/ Thanks alexliu!! I could get Composite Column working on SELECT statement, however got an exception when tried to write data into Composite Column using following hive script and custom UDF. Here is relevant log on Exception: <blockquote> [2013-02-06 22:23:15,252] FATAL {ExecReducer} - org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"blr_01","_col1":1350465782,"_col2":"WARN"},"value":{"_col0":1},"alias":0} at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: InvalidRequestException(why:Not enough bytes to read value of component 0) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:603) ... </blockquote> Here is my hive script: <pre><code>CREATE TEMPORARY FUNCTION setCompositeLongString AS 'com.cisco.iep.hive.plugins.WriteCompositeLongString'; INSERT OVERWRITE TABLE TestTable SELECT src_id, setCompositeLongString(cast(alert_time as BIGINT), severity), COUNT(alert_type) FROM alert_instance GROUP BY src_id, cast(alert_time as BIGINT), severity;</code></pre> And here is custom UDF to write data into Composite Column: <pre><code>package com.cisco.iep.hive.plugins; import java.io.ByteArrayOutputStream; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.hive.ql.exec.UDF; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; public class WriteCompositeLongString extends UDF{ public BytesWritable evaluate(final LongWritable l, final Text s) { List<byte[]> b = new ArrayList<byte[]>(2); b.add(longWritableToByteArray(l)); b.add(textToByteArray(s)); return new BytesWritable(makeComposite(b)); } private byte[] textToByteArray(Text s){ if(s==null){ return null; } return s.getBytes(); } private byte[] longWritableToByteArray(final LongWritable l){ if(l==null){ return null; } return ByteBuffer.allocate(Long.SIZE).putLong(l.get()).array(); } private byte[] makeComposite(final List<byte[]> b) { ByteArrayOutputStream bos = new ByteArrayOutputStream(); for (int i = 0; i < b.size(); i++) { bos.write((byte) ((b.get(i).length >> (7 + 1)) & 0xFF)); bos.write((byte) (b.get(i).length & 0xFF)); for (int j = 0; j < b.get(i).length; j++) { bos.write(b.get(i)[j] & 0xFF); } bos.write((byte) 0); } return bos.toByteArray(); } }</code></pre> Please advise. alexliu on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table#post-8747 Mon, 28 Jan 2013 20:24:56 +0000 alexliu 8747@http://www.datastax.com/support-forums/ non-cql composite row key is the same format as a composite column. But cql composite row key is in a little different format, so you should modify the UDF for cql composite row key. alexliu on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table#post-8746 Mon, 28 Jan 2013 19:24:07 +0000 alexliu 8746@http://www.datastax.com/support-forums/ you should be able to do the same to Composite Row Key, because that composite row key is stored the same format as composite column. Basically you retrieve the key as binary data, then use UDF to retrieve/parse the data. dipesh on "How do you use composite columns (from cassandra) in a hive table?" http://www.datastax.com/support-forums/topic/how-do-you-use-composite-columns-from-cassandra-in-a-hive-table#post-8593 Wed, 23 Jan 2013 06:38:08 +0000 dipesh 8593@http://www.datastax.com/support-forums/ I am wondering, will this technique work with Composite Row Key as well? So the table (CF) would look like this: create column family TestTable with key_validation_class = 'CompositeType (UTF8Type, UTF8Type)' and comparator = 'UTF8Type' ; Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2061 Fri, 08 Jun 2012 18:15:33 +0000 Anonymous 2061@http://www.datastax.com/support-forums/ When you start up all your nodes, use dsetool movejt to set the jobtracker at the specified datacenter / location. Probably the analytics node is not a seed, so it doesn't start up the jobtracker automatically. It should remember the setting. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2049 Wed, 06 Jun 2012 21:51:57 +0000 Anonymous 2049@http://www.datastax.com/support-forums/ We created a cluster across an ec2 availability zones within a region. We are using EC2Snitch and we are having to bring up the nodes one at a time. What would the proper procedure be fore bringing up a 3 cassandra node, one in each availability zone, then bringing up an analytical node, then possibly adding more nodes as needed. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2047 Wed, 06 Jun 2012 18:05:08 +0000 Anonymous 2047@http://www.datastax.com/support-forums/ How can you tell there is no jobtracker? If I run #dsetool jobtracker 10.10.21.85:8012 I see there is a (TT) in the ring i was wondering if that should be a (JT). If so how do you change that with service des start method. the contents of /etc/default/dse # enable this to start Hadoop's JobTracker and/or TaskTrackers on this machine. # If left disabled, this will act as a regular Cassandra node. HADOOP_ENABLED=1 # enable this to set the replication factor for CFS. Note that this will only # have an effect the first time a cluster is started with HADOOP_ENABLED=1 and # after that will be a no-op. Defaults to 1. #CFS_REPLICATION_FACTOR=1 # enable this to start Solr search indexing on this machine. If left disabled, # this will act as a regular Cassandra node. SOLR_ENABLED=0 ~ Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2046 Wed, 06 Jun 2012 16:25:45 +0000 Anonymous 2046@http://www.datastax.com/support-forums/ What happened to your jobtracker? From dsetool ring you don't seem to be running one. Try starting the analytics node with -j option. Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2045 Wed, 06 Jun 2012 13:49:37 +0000 Anonymous 2045@http://www.datastax.com/support-forums/ dsetool ring >>> Address DC Rack Workload Status State Load Owns Token 113427455640312821154458202477256070485 10.10.21.85 us-east 1a Analytics(tt) Up Normal 5.49 MB 33.33% 0 10.120.206.93 us-east 1b Cassandra Up Normal 5.22 MB 33.33% 56713727820156410577229101238628035242 10.68.78.182 us-east 1d Cassandra Up Normal 5.59 MB 33.33% 113427455640312821154458202477256070485 hive.log>> 2012-06-06 01:56:14,558 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(780)) - Got exception: java.io.IOException UnavailableException() 2012-06-06 01:56:14,562 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(783)) - java.io.IOException: UnavailableException() at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:686) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readINodeColumns(CassandraFileSystemThriftStore.java:617) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:569) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:561) at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.getFileStatus(CassandraFileSystem.java:508) at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:377) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1040) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$700(HiveMetaStore.java:143) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1081) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:362) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:400) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:540) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: UnavailableException() at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7764) at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570) at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.datastax.bdp.util.CassandraProxyClient.invoke(CassandraProxyClient.java:407) at $Proxy5.get_slice(Unknown Source) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:668) ... 30 more 2012-06-06 01:56:14,564 ERROR exec.Task (SessionState.java:printError(380)) - FAILED: Error in metadata: MetaException(message:Got exception: java.io.IOException UnavailableException()) org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException UnavailableException()) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:546) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3479) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: MetaException(message:Got exception: java.io.IOException UnavailableException()) at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:784) at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:385) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1040) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$700(HiveMetaStore.java:143) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1081) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:362) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1078) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:400) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:540) ... 17 more 2012-06-06 01:56:14,565 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2043 Wed, 06 Jun 2012 09:42:26 +0000 Anonymous 2043@http://www.datastax.com/support-forums/ What is your cluster configuration (dsetool ring)? Can you post a full exception stacktrace from hive logs? Anonymous on "Failure running Hive examples on DSE" http://www.datastax.com/support-forums/topic/failure-running-hive-examples-on-dse#post-2042 Wed, 06 Jun 2012 02:00:52 +0000 Anonymous 2042@http://www.datastax.com/support-forums/ It seems to be similar to this problem <a href="http://www.datastax.com/support-forums/topic/there-was-a-problem-with-the-cassandra-hive-metastore" rel="nofollow">http://www.datastax.com/support-forums/topic/there-was-a-problem-with-the-cassandra-hive-metastore</a> But the error is slightly different. hive> CREATE TABLE invites (foo INT, bar STRING) > PARTITIONED BY (ds STRING); FAILED: Error in metadata: MetaException(message:Got exception: java.io.IOException UnavailableException()) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask We are trying to run the lastest DSE up on amazon ec2. Any thoughts on our configuration error? Thanks, Todd Anonymous on "Cassandra Hive Storage Handler Sample for Map data type" http://www.datastax.com/support-forums/topic/cassandra-hive-storage-handler-sample-for-map-data-type#post-1957 Tue, 22 May 2012 07:23:19 +0000 Anonymous 1957@http://www.datastax.com/support-forums/ My use case is to have a Hive table definition with MAP Hive data type so that it can support varying set of key value properties per each row which is allowed in Cassandra. Some thing like what's given below. CREATE EXTERNAL TABLE mapExample (foo INT, bar STRING, properties MAP<STRING,STRING>) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ("cassandra.host" = "127.0.0.1","cassandra.port" = "9160","cassandra.ks.name" = "examples","cassandra.cf.name" = "map" ); But I am not sure how to provide the cassandra.column.mappings etc. since there is little documentation on how to use map data type with Hive, specially with Cassandra Storage handler. Any pointers regarding this are greatly appreciated. Regards Buddhika