hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From foolbear <foolbea...@gmail.com>
Subject Does YARN has hash-based shuffle plugin?
Date Thu, 15 Jan 2015 04:26:55 GMT
Hi

In YARN, shuffle and sort is pluggable:
http://hadoop.apache.org/docs/r2.5.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html

Currently, shuffle is based on sort. But many of my mapreduce jobs do not
need sort.
To improve performance, maybe it is better to avoid sort and use hash
instead.

So, is there a hash-based shuffle plugin?
Seems hadoop itselt does not do this. Any third-party implements?

Thanks

Mime
View raw message