Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

Mapping composite column names on Hive

(4 posts) (3 voices)

Started 8 months ago by pierantonio.merlino
Latest reply from dipesh
This topic is not a support question

Tags:

No tags yet.

pierantonio.merlino
Member
Hi,
I've to map Cassandra columns families on Hive tables.
Since I've dynamic fields in Cassandra, I use the following string
```
CREATE EXTERNAL TABLE Table1
(keyName string, columnName string, value string)
STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
WITH SERDEPROPERTIES ("cassandra.columns.mapping" = ":key,:column,:value" );
```
in order to create the external table on Hive. The result is a table where each row is as follows:

rowName1 columnName1 value1
rowName2 columnName2 value2
...

Since I use composite column names in Cassandra (Composite(columnNameA, columnNameB)), is it possible to map them into two different columns on Hive?
What I want is as follows:

rowName1 columnNameA1 columnNameB1 value1.

How can I map these columns on Hive?

Thanks in advance.
Posted 8 months ago #
Piotr Kołaczkowski
Administrator

There is no dedicated support in Hive for composite columns. In wide-row mode, it scans each row sequentially and fetches each column to produce a triple (row-key, column-name, value). Outputting as triples is hardcoded. Anyway, this could make a nice feature-request.

Posted 7 months ago #
pierantonio.merlino
Member

Thank you for the answer.
I solved the problem mapping the composite columns in to a struct in Hive and writing a custom serializer/deserializer.

Posted 7 months ago #

dipesh
Member

If possible, can you please share a sample code for serializer and deserializer?
And/or provide necessary pointers that could help in that regard.

Here is what I have, but not working.

package com.cisco.iep.hive.plugins;

import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;

public class WriteCompositeLongString extends UDF{

	public BytesWritable evaluate(final LongWritable l, final Text s) {

		List<byte[]> b = new ArrayList<byte[]>(2);
		b.add(longWritableToByteArray(l));
		b.add(textToByteArray(s));

		return new BytesWritable(makeComposite(b));
	}

	private byte[] textToByteArray(Text s){
		if(s==null){
			return null;
		}
		return s.getBytes();
	}

	private byte[] longWritableToByteArray(final LongWritable l){
		if(l==null){
			return null;
		}
		return ByteBuffer.allocate(Long.SIZE).putLong(l.get()).array();
	}

	private byte[] makeComposite(final List<byte[]> b) {
		ByteArrayOutputStream bos = new ByteArrayOutputStream();
		for (int i = 0; i < b.size(); i++) {
			bos.write((byte) ((b.get(i).length >> (7 + 1)) & 0xFF));
			bos.write((byte) (b.get(i).length & 0xFF));
			for (int j = 0; j < b.get(i).length; j++) {
				bos.write(b.get(i)[j] & 0xFF);
			}
			bos.write((byte) 0);
		}
		return bos.toByteArray();
	}

}

Mapping composite column names on Hive

Reply