Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com
 designates 64.18.0.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAFnReZbFc=foHA1TU=Cz7XUNUeobY6sb1HA1+dPF9-2HzpGvXg@mail.gmail.com>
References: 
 <CAFnReZbFc=foHA1TU=Cz7XUNUeobY6sb1HA1+dPF9-2HzpGvXg@mail.gmail.com>
Date: Thu, 28 Mar 2013 15:31:29 +0530
Message-ID: 
 <CAEAKFL8eaeVL_JPHU8yNERbTTyGxhEc4AG974sQwPfDreGXE6w@mail.gmail.com>
Subject: Re: Find reducer for a key
From: Hemanth Yamijala <yhemanth@thoughtworks.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=e89a8fb1fbba14fb2004d8f93d24

--e89a8fb1fbba14fb2004d8f93d24
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Not sure if I am answering your question, but this is the background. Every
MapReduce job has a partitioner associated to it. The default partitioner
is a HashPartitioner. You can as a user write your own partitioner as well
and plug it into the job. The partitioner is responsible for splitting the
map outputs key space among the reducers.

So, to know which reducer a key will go to, it is basically the value
returned by the partitioner's getPartition method. For e.g this is the code
in the HashPartitioner:

  public int getPartition(K2 key, V2 value,
                          int numReduceTasks) {
    return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
  }

mapred.task.partition is the key that defines the partition number of this
reducer.

I guess you can piece together these bits into what you'd want.. However, I
am interested in understanding why you want to know this ? Can you share
some info ?

Thanks
Hemanth


On Thu, Mar 28, 2013 at 2:17 PM, Alberto Cordioli <
cordioli.alberto@gmail.com> wrote:

> Hi everyone,
>
> how can i know the keys that are associated to a particular reducer in
> the setup method?
> Let's assume in the setup method to read from a file where each line
> is a string that will become a key emitted from mappers.
> For each of these lines I would like to know if the string will be a
> key associated with the current reducer or not.
>
> I read something about mapred.task.partition and mapred.task.id, but I
> didn't understand the usage.
>
>
> Thanks,
> Alberto
>
>
> --
> Alberto Cordioli
>

--e89a8fb1fbba14fb2004d8f93d24
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>Not sure if I am answering your que=
stion, but this is the background. Every MapReduce job has a partitioner as=
sociated to it. The default partitioner is a HashPartitioner. You can as a =
user write your own partitioner as well and plug it into the job. The parti=
tioner is responsible for splitting the map outputs key space among the red=
ucers.</div>
<div><br></div><div style>So, to know which reducer a key will go to, it is=
 basically the value returned by the partitioner&#39;s getPartition method.=
 For e.g this is the code in the HashPartitioner:</div><div style><br></div=
>
<div style><div>=A0 public int getPartition(K2 key, V2 value,</div><div>=A0=
 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 int numReduceTasks) {</div=
><div>=A0 =A0 return (key.hashCode() &amp; Integer.MAX_VALUE) % numReduceTa=
sks;</div><div>=A0 }</div><div>
<br></div><div style>mapred.task.partition is the key that defines the part=
ition number of this reducer.=A0</div><div style><br></div><div style>I gue=
ss you can piece together these bits into what you&#39;d want.. However, I =
am interested in understanding why you want to know this ? Can you share so=
me info ?</div>
<div style><br></div><div style>Thanks</div><div style>Hemanth</div></div><=
/div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu, =
Mar 28, 2013 at 2:17 PM, Alberto Cordioli <span dir=3D"ltr">&lt;<a href=3D"=
mailto:cordioli.alberto@gmail.com" target=3D"_blank">cordioli.alberto@gmail=
.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
how can i know the keys that are associated to a particular reducer in<br>
the setup method?<br>
Let&#39;s assume in the setup method to read from a file where each line<br=
>
is a string that will become a key emitted from mappers.<br>
For each of these lines I would like to know if the string will be a<br>
key associated with the current reducer or not.<br>
<br>
I read something about mapred.task.partition and <a href=3D"http://mapred.t=
ask.id" target=3D"_blank">mapred.task.id</a>, but I<br>
didn&#39;t understand the usage.<br>
<br>
<br>
Thanks,<br>
Alberto<br>
<br>
<br>
--<br>
Alberto Cordioli<br>
</blockquote></div><br></div>

--e89a8fb1fbba14fb2004d8f93d24--