hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@cloudera.com>
Subject Re: Anyway to sort "keys" before Reduce function in Hadoop ?
Date Mon, 15 Jun 2009 23:53:18 GMT
Hey Kun,

Keys given to a given reducer instance are given in sorted order.  Meaning,
for a given reducer JVM instance, the reduce function will be called several
times, once for each key.  The order in which the keys are given to the
reduce function are sorted.  The sorting happens in the shuffle phase, which
is basically partitioning and sorting.  That said, if you have one reducer
(which isn't possible in large jobs), keys will be given to you in sorted

You may be interested in the combiner phase, which is essentially a mini
reduce that happens before data is transferred between mapper and reducer:

<http://wiki.apache.org/hadoop/HadoopMapReduce> (grep for "combine")

You may also find these videos useful:

Hope this helps.  Let me know if I misunderstood your question.


On Mon, Jun 15, 2009 at 4:22 PM, Kunsheng Chen <keyek@yahoo.com> wrote:

> Hi everyone,
> Is there anyway to sort the "keys" before Reduce but after Map ?
> I also think of sorting keys myself in Reduce function, but it might take
> too many memory once the number of results getting large.
> I am thinking of using some numeric value as "keys" in Reduce (which was
> calculate by Map). If it is possible, I could output my results by some
> orders easily.
> Thanks in advance,
> -Kun

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message