http://www.datastax.com/support-forums/topic/mapreduce-spill-failure Software, Support, and Training for Apache Cassandra en-US Thu, 23 May 2013 21:38:33 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/mapreduce-spill-failure/page/2#post-1521 Tue, 03 Apr 2012 13:27:04 +0000 Anonymous 1521@http://www.datastax.com/support-forums/ <p>Nevermind. I see my error: MyReducer should be parameterized as <code>&lt;Text,IntWritable,Text,IntWritable&gt;</code> and then the reduce method should have the following signature:<br /> <code>public void reduce(Text key, Iterable&lt;IntWritable&gt; counts, Context context)</code></p> <p>Everything is working now. Thanks again for all the help. </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1518 Mon, 02 Apr 2012 20:19:46 +0000 Anonymous 1518@http://www.datastax.com/support-forums/ <p>That solved my spill problem. One last question and I'll be all set:</p> <p>My mapper runs fine now and the job succeeds. The output file, however, looks like it didn't aggregate the results (all counts are 1 and the same key shows up multiple times in the output file). My reducer and job configuration look exactly like the WordCount example as far as I can tell (included below). Can you see anything I'm doing wrong?</p> <pre><code>public static class MyReducer extends Reducer&lt;Text, Iterable&lt;IntWritable&gt;, Text, IntWritable&gt; { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable&lt;IntWritable&gt; counts, Context context) throws Exception { int sum = 0; for (IntWritable count : counts) { sum += count.get(); } result.set(sum); context.write(key, result); } }</code></pre> <p>And in the job configuration, I have:<br /> <pre><code>job.setCombinerClass(MyReducer.class); job.setReducerClass(MyReducer.class);</code></pre> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1517 Mon, 02 Apr 2012 19:49:19 +0000 tjake 1517@http://www.datastax.com/support-forums/ <p>col2.value().array() isn't safe (could generate a huge amount of garbage. use ByteBufferUtil.string(col2.value()), try that </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1516 Mon, 02 Apr 2012 19:35:01 +0000 Anonymous 1516@http://www.datastax.com/support-forums/ <p>Barely anything. Here's the map method in its entirety:</p> <pre><code>public void map(ByteBuffer key, SortedMap&lt;ByteBuffer, IColumn&gt; columns, Context context) throws IOException, InterruptedException { IColumn col1 = (IColumn)columns.get(ByteBuffer.wrap(&quot;column1&quot;.getBytes(&quot;UTF-8&quot;))); IColumn col2 =(IColumn) columns.get(ByteBuffer.wrap(&quot;column2&quot;.getBytes(&quot;UTF-8&quot;))); if(col1 != null &amp;&amp; col1.value()!=null &amp;&amp; col2 != null &amp;&amp; col2.value()!=null ){ long value = ByteBufferUtil.toLong(col1.value()); if(value &gt; 0 &amp;&amp; value &lt;= THRESHOLD){ Text outputKey = null; try{ outputKey = new Text(col2.value().array()); }catch(Exception e){ System.out.println(&quot;key is null&quot;); } if(outputKey != null){ context.write(outputKey,new IntWritable(1)); } } } }</code></pre> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1515 Mon, 02 Apr 2012 19:24:16 +0000 tjake 1515@http://www.datastax.com/support-forums/ <p>Yeah that's fine. I have no idea why 10G of spill is generated from 600M of data. What exactly is the mapper doing with the data? </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1514 Mon, 02 Apr 2012 19:19:34 +0000 Anonymous 1514@http://www.datastax.com/support-forums/ <p>As far as I can tell, I am setting that... in my Job configuration, I do the following:</p> <pre><code>List&lt;ByteBuffer&gt; colList = new ArrayList&lt;ByteBuffer&gt;(); colList.add(ByteBuffer.wrap(&quot;column1&quot;.getBytes(&quot;UTF-8&quot;))); colList.add(ByteBuffer.wrap(&quot;column2&quot;.getBytes(&quot;UTF-8&quot;))); SlicePredicate predicate = new SlicePredicate().setColumn_names(colList); //snip ConfigHelper.setInputSlicePredicate(job.getConfiguration(), predicate);</code></pre> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1512 Mon, 02 Apr 2012 17:24:53 +0000 tjake 1512@http://www.datastax.com/support-forums/ <p>Can you avoid range slicing? do you know the columns you need? if so you can set cassandra.slice.predicate.column_names </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1508 Mon, 02 Apr 2012 16:59:22 +0000 Anonymous 1508@http://www.datastax.com/support-forums/ <p>I thought I did, but upon re-working my Mapper, I'm still having issues. I don't understand why my mapper is spilling at all... my column family has a bunch of columns but in the slice predicate, I'm just looking at 2 of them. My mapper is quite simple: for each row, it looks at column1 and if the value is &gt; Threshold, it writes the value of column2 and an IntWritable set to 1 to the context. </p> <p>My reducer just sums up the ints and outputs the total for each key.</p> <p>Even though my test column family only contains 300 MB of data, approximately 10 GB of spill files are produced. I can see in the spill files that it contains not only the value of column2, but column1 as well (which I never write to the context). I thought the spill files should only contain the output of the map operation that doesn't fit in memory any more? </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1499 Mon, 02 Apr 2012 15:31:05 +0000 Stephanie 1499@http://www.datastax.com/support-forums/ <p>Hi Yossarian, </p> <p>Were you able to get help with everything? </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1490 Mon, 02 Apr 2012 13:46:57 +0000 Anonymous 1490@http://www.datastax.com/support-forums/ <p>I didn't think it was too small (it had 6 GB free and my column family on which I was running the job was only 600 MB). When I changed the directory to an explicit dir on a filesystem with more space, it worked. Apparently, my mapper has a bug and it's generating way too much output. </p> <p>Thanks for the help. </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1487 Mon, 02 Apr 2012 13:29:06 +0000 tjake 1487@http://www.datastax.com/support-forums/ <p>According to this your mapred.local.dir is ${hadoop.tmp.dir}/mapred/local</p> <p>Is your /tmp dir small?</p> <p>Can you try setting this to fixed directory in mapred-site.xml ? </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1486 Mon, 02 Apr 2012 13:12:38 +0000 Anonymous 1486@http://www.datastax.com/support-forums/ <p>Here's the job.xml:</p> <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;no&quot;?&gt;&lt;configuration&gt; &lt;property&gt;&lt;name&gt;fs.s3n.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.s3native.NativeS3FileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.cache.levels&lt;/name&gt;&lt;value&gt;2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.inode.mode&lt;/name&gt;&lt;value&gt;SYNC&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.tmp.dir&lt;/name&gt;&lt;value&gt;/tmp/hadoop-${user.name}&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.native.lib&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;map.sort.class&lt;/name&gt;&lt;value&gt;org.apache.hadoop.util.QuickSort&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;dse.cfs.repair.batchsize&lt;/name&gt;&lt;value&gt;300&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.client.idlethreshold&lt;/name&gt;&lt;value&gt;4000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.thrift.address&lt;/name&gt;&lt;value&gt;nyarchapp01d&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.system.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/mapred/system&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.persist.jobstatus.hours&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.skip.checksum.errors&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.default.name&lt;/name&gt;&lt;value&gt;cfs:///&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.cluster.reduce.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reducer.new-api&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.child.tmp&lt;/name&gt;&lt;value&gt;./tmp&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.har.impl.disable.cache&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.partitioner.class&lt;/name&gt;&lt;value&gt;org.apache.cassandra.dht.RandomPartitioner&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.skip.reduce.max.skip.groups&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.heartbeats.in.second&lt;/name&gt;&lt;value&gt;100&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.dns.nameserver&lt;/name&gt;&lt;value&gt;default&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.sort.factor&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.input.columnfamily&lt;/name&gt;&lt;value&gt;MyEvent&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.timeout&lt;/name&gt;&lt;value&gt;600000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.max.tracker.failures&lt;/name&gt;&lt;value&gt;4&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.rpc.socket.factory.class.default&lt;/name&gt;&lt;value&gt;org.apache.hadoop.net.StandardSocketFactory&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.jobhistory.lru.cache.size&lt;/name&gt;&lt;value&gt;5&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.hdfs.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.hdfs.DistributedFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.queue.default.acl-administer-jobs&lt;/name&gt;&lt;value&gt;*&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.key.class&lt;/name&gt;&lt;value&gt;org.apache.hadoop.io.Text&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.skip.map.auto.incr.proc.count&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.complete.cancel.delegation.tokens&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.mapfile.bloom.size&lt;/name&gt;&lt;value&gt;1048576&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.reduce.shuffle.connect.timeout&lt;/name&gt;&lt;value&gt;180000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.blacklist.fault-timeout-window&lt;/name&gt;&lt;value&gt;180&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;tasktracker.http.threads&lt;/name&gt;&lt;value&gt;40&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.shuffle.merge.percent&lt;/name&gt;&lt;value&gt;0.66&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.inputformat.class&lt;/name&gt;&lt;value&gt;org.apache.cassandra.hadoop.ColumnFamilyInputFormat&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.ftp.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.ftp.FTPFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.compress&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.bytes.per.checksum&lt;/name&gt;&lt;value&gt;512&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.combine.recordsBeforeProgress&lt;/name&gt;&lt;value&gt;10000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.healthChecker.script.timeout&lt;/name&gt;&lt;value&gt;600000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;topology.node.switch.mapping.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.net.ScriptBasedMapping&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.thrift.port&lt;/name&gt;&lt;value&gt;9160&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reduce.slowstart.completed.maps&lt;/name&gt;&lt;value&gt;0.05&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reduce.max.attempts&lt;/name&gt;&lt;value&gt;4&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.input.keyspace&lt;/name&gt;&lt;value&gt;ReportingKeyspace&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.ramfs.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.InMemoryFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.skip.map.max.skip.records&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.cluster.map.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.local.block.size&lt;/name&gt;&lt;value&gt;67108864&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.security.group.mapping&lt;/name&gt;&lt;value&gt;org.apache.hadoop.security.ShellBasedUnixGroupsMapping&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.persist.jobstatus.dir&lt;/name&gt;&lt;value&gt;/jobtracker/jobsInfo&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jar&lt;/name&gt;&lt;value&gt;cfs:/tmp/hadoop-apparch/mapred/staging/apparch/.staging/job_201203301707_0004/job.jar&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.s3.buffer.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/s3&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;job.end.retry.attempts&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;dse.consistencylevel.read&lt;/name&gt;&lt;value&gt;ONE&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.file.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.LocalFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.local.dir.minspacestart&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.compression.type&lt;/name&gt;&lt;value&gt;RECORD&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;cassandra.input.predicate&lt;/name&gt;&lt;value&gt;0f00010b000000060000000c6f66666572696e67546b65790000000a646576696365436f64650000000d616666696c69617465436f64650000000b73657276696365546b65790000001373746172744576656e7454696d657374616d7000000015656e64456c6170736564506c61795365636f6e647300&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;dfs.permissions&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;topology.script.number.args&lt;/name&gt;&lt;value&gt;100&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.mapfile.bloom.error.rate&lt;/name&gt;&lt;value&gt;0.005&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.cluster.max.reduce.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.max.tracker.blacklists&lt;/name&gt;&lt;value&gt;4&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.profile.maps&lt;/name&gt;&lt;value&gt;0-2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.userlog.retain.hours&lt;/name&gt;&lt;value&gt;24&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.persist.jobstatus.active&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.security.authorization&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;local.cache.size&lt;/name&gt;&lt;value&gt;10737418240&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.min.split.size&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.map.tasks&lt;/name&gt;&lt;value&gt;3&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.cfs.impl&lt;/name&gt;&lt;value&gt;com.datastax.bdp.hadoop.cfs.CassandraFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.child.java.opts&lt;/name&gt;&lt;value&gt;-Xmx256m&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.counters.limit&lt;/name&gt;&lt;value&gt;120&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.value.class&lt;/name&gt;&lt;value&gt;org.apache.hadoop.io.SortedMapWritable&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;dse.autoconfigure.node&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.queue.name&lt;/name&gt;&lt;value&gt;default&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.retiredjobs.cache.size&lt;/name&gt;&lt;value&gt;1000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.server.listen.queue.size&lt;/name&gt;&lt;value&gt;128&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;job.end.retry.interval&lt;/name&gt;&lt;value&gt;30000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.inmem.merge.threshold&lt;/name&gt;&lt;value&gt;1000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.skip.attempts.to.start.skipping&lt;/name&gt;&lt;value&gt;2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.tasktracker.outofband.heartbeat.damper&lt;/name&gt;&lt;value&gt;1000000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.checkpoint.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/dfs/namesecondary&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reduce.tasks&lt;/name&gt;&lt;value&gt;1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.merge.recordsBeforeProgress&lt;/name&gt;&lt;value&gt;10000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.userlog.limit.kb&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.reduce.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;webinterface.private.actions&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.security.token.service.use_ip&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.sort.spill.percent&lt;/name&gt;&lt;value&gt;0.80&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.shuffle.input.buffer.percent&lt;/name&gt;&lt;value&gt;0.70&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.name&lt;/name&gt;&lt;value&gt;offering count&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.map.tasks.speculative.execution&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.util.hash.type&lt;/name&gt;&lt;value&gt;murmur&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.map.max.attempts&lt;/name&gt;&lt;value&gt;4&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.acl-view-job&lt;/name&gt;&lt;value&gt; &lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.handler.count&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.reduce.shuffle.read.timeout&lt;/name&gt;&lt;value&gt;180000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.expiry.interval&lt;/name&gt;&lt;value&gt;600000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.maxtasks.per.job&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.job.history.block.size&lt;/name&gt;&lt;value&gt;3145728&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;keep.failed.task.files&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.profile.reduces&lt;/name&gt;&lt;value&gt;0-2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.client.tcpnodelay&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.compression.codec&lt;/name&gt;&lt;value&gt;org.apache.hadoop.io.compress.DefaultCodec&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.map.index.skip&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.working.dir&lt;/name&gt;&lt;value&gt;cfs:/user/apparch&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.server.tcpnodelay&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.blacklist.fault-bucket-width&lt;/name&gt;&lt;value&gt;15&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.used.genericoptionsparser&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.mapper.new-api&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.map.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.logfile.size&lt;/name&gt;&lt;value&gt;10000000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reduce.tasks.speculative.execution&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.dir&lt;/name&gt;&lt;value&gt;cfs:/tmp/hadoop-apparch/mapred/staging/apparch/.staging/job_201203301707_0004&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.tasktracker.outofband.heartbeat&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.reduce.input.limit&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.security.authentication&lt;/name&gt;&lt;value&gt;simple&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.checkpoint.period&lt;/name&gt;&lt;value&gt;3600&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.reuse.jvm.num.tasks&lt;/name&gt;&lt;value&gt;24&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.completeuserjobs.maximum&lt;/name&gt;&lt;value&gt;100&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;jobtracker.thrift.address&lt;/name&gt;&lt;value&gt;0.0.0.0:9290&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.tracker.task-controller&lt;/name&gt;&lt;value&gt;org.apache.hadoop.mapred.DefaultTaskController&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.s3.maxRetries&lt;/name&gt;&lt;value&gt;4&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.cluster.max.map.memory.mb&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.reduce.shuffle.maxfetchfailures&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.acl-modify-job&lt;/name&gt;&lt;value&gt; &lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.local.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/mapred/local&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.hftp.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.hdfs.HftpFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.trash.interval&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.s3.sleepTimeSeconds&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.submit.replication&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.har.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.HarFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.map.output.compression.codec&lt;/name&gt;&lt;value&gt;com.datastax.bdp.hadoop.compression.SnappyCodec&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.dns.interface&lt;/name&gt;&lt;value&gt;default&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker&lt;/name&gt;&lt;value&gt;${dse.job.tracker}&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;dse.consistencylevel.write&lt;/name&gt;&lt;value&gt;ONE&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.submithost&lt;/name&gt;&lt;value&gt;nyarchapp01d&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.seqfile.sorter.recordlimit&lt;/name&gt;&lt;value&gt;1000000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.line.input.format.linespermap&lt;/name&gt;&lt;value&gt;1&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.taskScheduler&lt;/name&gt;&lt;value&gt;org.apache.hadoop.mapred.JobQueueTaskScheduler&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.webhdfs.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.hdfs.web.WebHdfsFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.local.dir.minspacekill&lt;/name&gt;&lt;value&gt;0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.sort.record.percent&lt;/name&gt;&lt;value&gt;0.05&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.reduce.class&lt;/name&gt;&lt;value&gt;com.myorg.hadoop.MyEventReducer&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.kfs.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.kfs.KosmosFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.temp.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/mapred/temp&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.reduce.tasks.maximum&lt;/name&gt;&lt;value&gt;2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.checkpoint.edits.dir&lt;/name&gt;&lt;value&gt;${fs.checkpoint.dir}&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.tasks.sleeptime-before-sigkill&lt;/name&gt;&lt;value&gt;5000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.reduce.input.buffer.percent&lt;/name&gt;&lt;value&gt;0.0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.indexcache.mb&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.plugins&lt;/name&gt;&lt;value&gt;com.datastax.bdp.hadoop.thriftfs.ThriftJobTrackerPlugin&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.split.metainfo.maxsize&lt;/name&gt;&lt;value&gt;10000000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.skip.reduce.auto.incr.proc.count&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.logfile.count&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.job.submithostaddress&lt;/name&gt;&lt;value&gt;10.40.199.130&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.seqfile.compress.blocksize&lt;/name&gt;&lt;value&gt;1048576&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.s3.block.size&lt;/name&gt;&lt;value&gt;67108864&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.taskmemorymanager.monitoring-interval&lt;/name&gt;&lt;value&gt;5000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.queue.default.state&lt;/name&gt;&lt;value&gt;RUNNING&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.acls.enabled&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.jobtracker.staging.root.dir&lt;/name&gt;&lt;value&gt;${hadoop.tmp.dir}/mapred/staging&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.queue.names&lt;/name&gt;&lt;value&gt;default&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.hsftp.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.hdfs.HsftpFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.tracker.http.address&lt;/name&gt;&lt;value&gt;0.0.0.0:50060&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.combine.class&lt;/name&gt;&lt;value&gt;com.myorg.hadoop.MyEventReducer&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.cfs-archive.impl&lt;/name&gt;&lt;value&gt;com.datastax.bdp.hadoop.cfs.CassandraFileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.reduce.parallel.copies&lt;/name&gt;&lt;value&gt;5&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.seqfile.lazydecompress&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.output.dir&lt;/name&gt;&lt;value&gt;/tmp/hadoopout&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.sort.mb&lt;/name&gt;&lt;value&gt;100&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.client.connection.maxidletime&lt;/name&gt;&lt;value&gt;10000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.compress.map.output&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;hadoop.security.uid.cache.secs&lt;/name&gt;&lt;value&gt;14400&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.tracker.report.address&lt;/name&gt;&lt;value&gt;127.0.0.1:0&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.healthChecker.interval&lt;/name&gt;&lt;value&gt;60000&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.client.kill.max&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;ipc.client.connect.max.retries&lt;/name&gt;&lt;value&gt;10&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapreduce.map.class&lt;/name&gt;&lt;value&gt;com.myorg.hadoop.MyEventMapper&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.s3.impl&lt;/name&gt;&lt;value&gt;org.apache.hadoop.fs.s3.S3FileSystem&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.user.jobconf.limit&lt;/name&gt;&lt;value&gt;5242880&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.job.tracker.http.address&lt;/name&gt;&lt;value&gt;0.0.0.0:50030&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.file.buffer.size&lt;/name&gt;&lt;value&gt;4096&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.jobtracker.restart.recover&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.serializations&lt;/name&gt;&lt;value&gt;org.apache.hadoop.io.serializer.WritableSerialization&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.task.profile&lt;/name&gt;&lt;value&gt;false&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;jobclient.output.filter&lt;/name&gt;&lt;value&gt;FAILED&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;mapred.tasktracker.map.tasks.maximum&lt;/name&gt;&lt;value&gt;2&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;io.compression.codecs&lt;/name&gt;&lt;value&gt;org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.checkpoint.size&lt;/name&gt;&lt;value&gt;67108864&lt;/value&gt;&lt;/property&gt; &lt;property&gt;&lt;name&gt;fs.local.subblock.size&lt;/name&gt;&lt;value&gt;2097152&lt;/value&gt;&lt;/property&gt; &lt;/configuration&gt;</code></pre> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1481 Fri, 30 Mar 2012 21:17:25 +0000 tjake 1481@http://www.datastax.com/support-forums/ <p>Can you post the job.xml file generated for your job? </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1480 Fri, 30 Mar 2012 21:10:47 +0000 Anonymous 1480@http://www.datastax.com/support-forums/ <p>I set that property in both $DSE_HOME/resources/hadoop/conf/taskcontroller.cfg and added the xml property declaration to $DSE_HOME/resources/hadoop/conf/dse-core.xml but I still get the same error. </p> http://www.datastax.com/support-forums/topic/mapreduce-spill-failure#post-1479 Fri, 30 Mar 2012 21:00:10 +0000 tjake 1479@http://www.datastax.com/support-forums/ <p>Sounds like you need to setup your mapred.local.dir on the nodes. </p> <p> &lt;property&gt;<br /> &lt;name&gt;mapred.local.dir&lt;/name&gt;<br /> &lt;value&gt;/d1/mapred/local,/d2/mapred/local,/d3/mapred/local,/d4/mapred/local&lt;/value&gt;<br /> &lt;/property&gt;</p> <p>This defaults to /tmp/hadoop-${user.name} </p>