Number of Map Tasks

(6 posts) (2 voices)

• Started 1 year ago by
• Latest reply from tjake
• This topic is not resolved

Tags:

No tags yet.

1. 

   Anonymous
   Unregistered

   While using DSE 2.0, how can we get hadoop/hive to increase the number of map tasks for maximum performance? We are trying to run map reduce with the input data coming from cassandra. Are there things like cassandra input buffer size or blocks that map reduce can see and increase the number of map tasks?

   Posted 1 year ago #
2. 

   tjake
   Moderator

   Yes, The setting is "cassandra.input.split.size" its default is 64k rows

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   Thanks. Where do we set this parameter? In mapred-site.xml?

   Posted 1 year ago #
4. 

   tjake
   Moderator

   You can set it there or as part of the job using -D in hadoop cmd

   Posted 1 year ago #
5. 

   Anonymous
   Unregistered

   Is it the same as setting HIVE_OPT while invoking hive? Or should we make it part of TBLPROPERTIES while defining an external table?

   export HIVE_OPT="-Dcassandra.input.split.size=268435456"

   we are trying to increase the number of map tasks that get kicked of as a result of a simple query like

   select count(*) from tablefoo;

   Posted 1 year ago #
6. 

   tjake
   Moderator

   Make it part of the TBLPROPERTIES

   The smaller the split size is the more mappers it will create.

   Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.