Support Forums : DataStax

Suggestions for moving .har files in HDFS to CFS

(3 posts) (2 voices)

Tags:

No tags yet.

matt
Member

Please pardon the cross-post with the google group, but I could really use a suggestion or two on this problem.

I have data in .har files on a CDH2 Hadoop cluster. I'd like to transfer the contents of the .har archive into CFS. It appears as though hftp doesn't have the ability to read .har files. As far as I can tell that means I have to copy them to local storage, and then upload them. Does anybody have any ideas for how to make this work using distcp? distcp works great with hftp to copy non-.har files, but the .har files are a no-go.

Thanks,
Matt

Posted 2 years ago #
tjake
Moderator

Hmm I don't really know... You can try asking on #datastax-brisk on freenode irc

Posted 2 years ago #
matt
Member

OK, my solution for this is to first copy the .har files to CFS using hftp from the source cluster. Then I do a $BRISK_HOME/bin/brisk hadoop cp 'har://cfs-brisk000:9160/stuff/my.har/*' /user/cassandra/exploded_har

Now I have all of the files from the .har in /user/cassandra/exploded_har and can delete the source .har. This is going to result in one hell of a compaction, but at least I can plan for it.

I suspect that if I had first upgraded the source cluster to hadoop 20.203, I'd have been able to do a regular "har://hdfs-" style distcp from the source cluster.

Posted 2 years ago #

You must log in to post.