hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sam liu <samliuhad...@gmail.com>
Subject Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?
Date Wed, 16 Oct 2013 02:02:54 GMT
Hi Experts,

In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as
its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'.
However, seems Yarn did not execute the methods of
TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as
below:

Test 1: Add some code in the method readPartitions() and setConf() in
TeraSort#TotalOrderPartitioner to print some words and write some word to a
file.
Expected Result: Some words should be printed and wrote into a file
Actual Result: No word was printed and wrote into a file at all

Test 2: Remove all existing methods in TeraSort#TotalOrderPartitioner, but
only remaining some necessary but empty methods in it
Expected Result: TeraSort job will ocurr some exception, as the specified
Partitioner is not implemented at all
Actual Result: TeraSort job completed successfully without any exception

Above tests confused me a lot, because seems Yarn never use specified
partitioner TeraSort#TotalOrderPartitioner at all during job execution.

Any one can help provide the reasons?

Thanks very much!

Mime
View raw message