hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?
Date Fri, 18 Oct 2013 21:03:28 GMT
Apologies for the late response.

In hadoop-2 TeraSort uses the new org.apache.hadoop.mapreduce apis (not org.apache.hadoop.mapred).

Did you fiddle with the right TotalOrderPartitioner i.e. org.apache.hadoop.mapreduce.TotalOrderPartitioner?


On Oct 17, 2013, at 8:12 PM, sam liu <samliuhadoop@gmail.com> wrote:

> It's really weird and confusing me. Anyone can help this question? 
> Thanks!
> 2013/10/16 sam liu <samliuhadoop@gmail.com>
> Hi Experts,
> In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as its Partitioner:
'job.setPartitionerClass(TotalOrderPartitioner.class);'. However, seems Yarn did not execute
the methods of TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as below:
> Test 1: Add some code in the method readPartitions() and setConf() in TeraSort#TotalOrderPartitioner
to print some words and write some word to a file.
> Expected Result: Some words should be printed and wrote into a file
> Actual Result: No word was printed and wrote into a file at all
> Test 2: Remove all existing methods in TeraSort#TotalOrderPartitioner, but only remaining
some necessary but empty methods in it
> Expected Result: TeraSort job will ocurr some exception, as the specified Partitioner
is not implemented at all
> Actual Result: TeraSort job completed successfully without any exception
> Above tests confused me a lot, because seems Yarn never use specified partitioner TeraSort#TotalOrderPartitioner
at all during job execution. 
> Any one can help provide the reasons?
> Thanks very much!

Arun C. Murthy
Hortonworks Inc.

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message