Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of amits@infolinks.com
 designates 207.126.144.149 as permitted sender)
MIME-Version: 1.0
Date: Sun, 12 Jan 2014 18:25:58 +0200
Message-ID: 
 <CAAMYKhrbnXbEwaFd+on_BPDXjeL_ycPNopABnG7M+8byuubJzQ@mail.gmail.com>
Subject: manipulating key in combine phase
From: Amit Sela <amits@infolinks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae93403390f3dda04efc86ad1

--14dae93403390f3dda04efc86ad1
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I was wondering if it is possible to manipulate the key during combine:

Say I have a mapreduce job where the key has many qualifiers.
I would like to "split" the key into two (or more) keys if it has more
than, say 100 qualifiers.
In the combiner class I would do something like:

int count = 0;
for (Writable value: values) {
  if (++count >= 100){
    context.write(newKey, value);
  } else {
    context.write(key, value);
  }
}

where newKey is something like key+randomUUID

I know that the combiner can be called "zero, once or more..." and I'm
getting strange results (same key written more then once) so I would be
glad to get some deeper insight into how the combiner works.

Thanks,

Amit.

--14dae93403390f3dda04efc86ad1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,=A0<div><br></div><div>I was wondering if it is pos=
sible to manipulate the key during combine:</div><div><br></div><div>Say I =
have a mapreduce job where the key has many qualifiers.=A0</div><div>I woul=
d like to &quot;split&quot; the key into two (or more) keys if it has more =
than, say 100 qualifiers.</div>
<div>In the combiner class I would do something like:</div><div><br></div><=
div>int count =3D 0;</div><div>for (Writable value: values) {</div><div>=A0=
 if (++count &gt;=3D 100){</div><div>=A0 =A0 context.write(newKey, value);<=
/div><div>
=A0 } else {</div><div>=A0 =A0 context.write(key, value);</div><div>=A0 }</=
div><div>}</div><div><br></div><div>where newKey is something like key+rand=
omUUID</div><div><br></div><div>I know that the combiner can be called &quo=
t;zero, once or more...&quot; and I&#39;m getting strange results (same key=
 written more then once) so I would be glad to get some deeper insight into=
 how the combiner works.</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Amit.</div></div>

--14dae93403390f3dda04efc86ad1--