http://www.datastax.com/support-forums/topic/use-lzo-compressed-file-in-s3-as-the-data-source-for-hive-tables Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 10:23:53 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/use-lzo-compressed-file-in-s3-as-the-data-source-for-hive-tables#post-443 Mon, 15 Aug 2011 23:15:25 +0000 spencerho 443@http://www.datastax.com/support-forums/ <p>We have all our data files stored in S3 lzo compressed. Our current hadoop/hive cluster uses and produces this type of files.</p> <p>We now in the process evaluating whether brisk/cassandra can work with out existing data. I created an external table pointing to S3</p> <pre><code>create external table exp_web_xxx ( request STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY &#039;9&#039; LINES TERMINATED BY &#039;10&#039; STORED AS TEXTFILE LOCATION &#039;s3n://......&#039;;</code></pre> <p>to see if brisk/hive can get the data alright. We are using demo AMI for our brisk cluster. I changed the hadoop configuration according to</p> <blockquote><p><a href="https://github.com/riptano/brisk/wiki/Installing-LZO-compression" rel="nofollow">https://github.com/riptano/brisk/wiki/Installing-LZO-compression</a> </p></blockquote> <p>Though the demo AMI has a different path setup than the production brisk image, I believe I have changed brisk-env.sh and hadoop's mapred-site.xml accordingly. I also changed brisk/resources/hive/conf/hive-site.xml for lzo compression. However, I can't get hive to work with our lzo compressed files. When I performed some basic operation like <code></code><code>select * from exp_web_xxx limit 10</code> the output is not decompressed. </p> <p>Does anyone has experience running brisk/cassandra with lzo compressed files as input data? </p>