hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: the same key in different reducers
Date Wed, 09 Jun 2010 22:15:37 GMT
I think the question was about the mapper keys and their assignment to
reducers.  The Parititioner<K,V> API looks like:

public abstract int getPartition(KEY key, VALUE value, int numPartitions);

So I assume it is entirely possible to write a partitioner that distributes
the same key to multiple reducers and it does not have to be
non-deterministic.  It can assign the partition based on the value.

Is this correct?

On Wed, Jun 9, 2010 at 2:43 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> I think his question was about the output key from reducer.
>
> On Wed, Jun 9, 2010 at 2:40 PM, James Seigel <james@tynt.com> wrote:
>
> > Oleg,
> >
> > Are you wanting to have them in different reducers?  If so then you can
> > write a Comparable object to make that happen.
> >
> > If you want them to be on the same reducer, then that is what hadoop will
> > do.
> >
> > :)
> >
> >
> > On 2010-06-09, at 3:06 PM, Ted Yu wrote:
> >
> > > Can you disclose more about how K3 is generated.
> > > From your description below, it is possible.
> > >
> > > On Wed, Jun 9, 2010 at 1:17 AM, Oleg Ruchovets <oruchovets@gmail.com>
> > wrote:
> > >
> > >> Hi ,
> > >> My hadoop job writes results of map/reduce to HBase.
> > >> I have 3 reducers.
> > >>
> > >> Here is a sequence of input and output parameters for Mapper ,
> Combiner
> > and
> > >> Reducer
> > >>   *input: InputFormat<K1,V1>
> > >>   mapper: Mapper<K1,V1,K2,V2>
> > >>   combiner: Reducer<K2,V2,K2,V2>
> > >>   reducer: Reducer<K2,V2,K3,V3>
> > >>   output: RecordWriter<K3,V3>
> > >>
> > >> *My question:
> > >> Is it possible that more than one reducer has the same output key K3.
> > >> Meaning in case I have 3 reducers is it possible that
> > >> reducer1    K3 -* 1* , V3 [1,2,3]
> > >> reducer2    K3 - 2 , V3 [5,6,9]
> > >> reducer3    K3 - *1* , V3 [10,15,22]
> > >>
> > >> As you can see reducer1 has K3 - 1 and reducer3 has K3 - 1.
> > >> So is that case possible or every and every reducer has unique output
> > key?
> > >>
> > >> Thanks in advance
> > >> Oleg.
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message