Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

brisk does not appear to use snappy "by default"

(5 posts) (3 voices)

Started 1 year ago by SVBridget
Latest reply from SVBridget
This topic is resolved

Tags:

nakliyat

SVBridget
Member

Hello-
The documentation for brisk beta-2 says "BRISK-207: New Snappy Compression Codec built on Google Snappy is now used internally for automatic CassandraFS block compression."

It appears that Snappy is never used "automatically" by CFS though. However, if we do a file put, the file remains the same size in cfs. I would expect to see a smaller file than the original file size. Similarly, for a hive table that is not using the cassandrastoragehandler (but is using cfs to store its data, since cfs backs hive in brisk) - we're also seeing that the data is not compressed automatically, either. We have to set some hive parameters to do this:
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=com.hadoop.compression.snappy.SnappyCodec;
SET mapred.output.compression.type=BLOCK;

I'm not sure if we are interpreting the word "automatic" incorrectly in the description of BRISK-207, or if we're doing something wrong, or there's a bug. I guess I was expecting that snappy would be compressing *everything* we put into cfs, no matter what route we take. Is there a way to make cfs behave this way all the time? Or is this not recommended?

Any more guidance you can provide would be appreciated...

Thanks!
-B

Posted 1 year ago #
tjake
Moderator

When you say file is the same size you mean as reported by hadoop ls ?
Internally on disk the dfs blocks are compressed but hadoop thinks it's a regular uncompressed block

Posted 1 year ago #
SVBridget
Member

Here's our scenario:
* A: We have a text file that is 8 MB (as reported by linux, e.g. before it is in hadoop).
* B: When we do a brisk hadoop put of this file, then get the file size from hadoop, it's also 8 MB
* C: We then made a hive table that points to the raw file that we put into hadoop, and then we selected out of that hive table and into a new hive table, and then we did a brisk hadoop fs -ls on the underlying data file for that 2nd hive table, and it was 8 MB
* D: Finally, we did basically the same thing, except we set hive properties to specify that we should do compression (that's the three SET statements in my original post). Then we did a brisk hadoop fs -ls on the underlying file for the 2nd hive table, and it's 2 MB

So this is basically what led to my questions... this seems pretty confusing. If CFS is compressing "automatically", I would expect the size of the underlying data file for case "C" to be the same as for "D". It seems like you are saying that hadoop is going to report the uncompressed size, and if that is the case, shouldn't "C" and "D" both report 8 MB?

Sorry if it seems like we're going into a rat hole on this, but we really would like to understand how this works, so we can make sure we are compressing properly...

And at a nuts and bolts level, our question is, do we need to specify any compression parameters to hive (e.g. the three SET statements in my original post), OR since CFS snappy compresses "everything", we can simply skip any hive compression settings, and our data that we put in any hive table will be compressed.

Thanks!
-B

Posted 1 year ago #
SVBridget
Member

Follow-up from the IRC...

<@tjake> your last post is correct
<@tjake> hadoop reports 8MB but on disk it's 2MB
<SVBridget> is there a way to ask hadoop how much space it is taking up on disk?
<@tjake> no but you can ask cassandra :)
<SVBridget> it's weird that for D it doesn't say 8MB also...
<@tjake> well in case of D hadoop knows its being compressed
<@tjake> in CFS we are compressing below the hadoop api
<@tjake> so it doesn't know
<SVBridget> ah ok
<SVBridget> that makes sense
<SVBridget> tx for clarifying, we'll take a look at what cassandra is saying for these files
<SVBridget> but basically there is no point in pre-compressing any files we are going to put into cassandra... because they're going to be stored block/snappy compressed anyway
<SVBridget> so it's pointless to, say, snappy compress, then put into cfs, right?
<@tjake> SVBridget: right
<@tjake> it does help to compress map output
<@tjake> the nice thing about the built-in snappy compression is its done task side
<@tjake> so lowers network io
<jeromatron> tjake: cool, so intermediate results are more efficiently compressed.
<@tjake> jeromatron: yes if you enable map output compression using snappy codec
<@tjake> the map output goes to hadoop.tmp which is on local disk
<jeromatron> sweet corn. yep - pretty cool. and you don't have to use lzo :)

Posted 1 year ago #
Anonymous
Unregistered

I've been a member of ankara nakliyat the new forum site, I found your site by chance while visiting the internet site I believe is a very useful forum to evden eve nakliyat ankara share useful information for all of us will benefit all ankara evden eve nakliyat şirketi hakkında genel bilgiler ve ankara nakliyat hakkında genel görüşlerin yer almış oldugu kaliteli evden eve nakliyat sitesine ait net portal bilgileri.those who labor to the forum site function ankara evden eve nakliyat family.
Mary Lou King

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.