Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of shahab.yunus@gmail.com
 designates 209.85.214.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPZM1Te5sAKwso+bYR2z8hZc0Rg-KuOoidiO9+hgXCbOKrLO0w@mail.gmail.com>
References: 
 <CAPZM1Te5sAKwso+bYR2z8hZc0Rg-KuOoidiO9+hgXCbOKrLO0w@mail.gmail.com>
Date: Fri, 30 Aug 2013 21:38:18 -0400
Message-ID: 
 <CAEo-6+RMCo6VAd4_GdLWYEmna1xFEDs15e2JvgzKcBz6nvevKQ@mail.gmail.com>
Subject: Re: Job config before read fields
From: Shahab Yunus <shahab.yunus@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11334a3cc3788304e53464ea

--001a11334a3cc3788304e53464ea
Content-Type: text/plain; charset=ISO-8859-1

I think you have to override/extend the Comparator to achieve that,
something like what is done in Secondary Sort?

Regards,
Shahab


On Fri, Aug 30, 2013 at 9:01 PM, Adrian CAPDEFIER <chivas314159@gmail.com>wrote:

> Howdy,
>
> I apologise for the lack of code in this message, but the code is fairly
> convoluted and it would obscure my problem. That being said, I can put
> together some sample code if really needed.
>
> I am trying to pass some metadata between the map & reduce steps. This
> metadata is read and generated in the map step and stored in the job
> config. It also needs to be recreated on the reduce node before the key/
> value fields can be read in the readFields function.
>
> I had assumed that I would be able to override the Reducer.setup()
> function and that would be it, but apparently the readFields function is
> called before the Reducer.setup() function.
>
> My question is what is any (the best) place on the reduce node where I can
> access the job configuration/ context before the readFields function is
> called?
>
> This is the stack trace:
>
>         at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:103)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>

--001a11334a3cc3788304e53464ea
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I think you have to override/extend the Comparator to achi=
eve that, something like what is done in Secondary Sort?<div><br></div><div=
>Regards,</div><div>Shahab</div></div><div class=3D"gmail_extra"><br><br><d=
iv class=3D"gmail_quote">
On Fri, Aug 30, 2013 at 9:01 PM, Adrian CAPDEFIER <span dir=3D"ltr">&lt;<a =
href=3D"mailto:chivas314159@gmail.com" target=3D"_blank">chivas314159@gmail=
.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr"><div><div><div>Howdy,<br><br></div><div>I apologise for th=
e lack of code in this message, but the code is fairly convoluted and it wo=
uld obscure my problem. That being said, I can put together some sample cod=
e if really needed.<br>

</div><div><br></div>I am trying to pass some metadata between the map &amp=
; reduce steps. This metadata is read and generated in the map step and sto=
red in the job config. It also needs to be recreated on the reduce node bef=
ore the key/ value fields can be read in the readFields function.<br>

<br>I had assumed that I would be able to override the Reducer.setup() func=
tion and that would be it, but apparently the readFields function is called=
 before the Reducer.setup() function.<br><br></div>My question is what is a=
ny (the best) place on the reduce node where I can access the job configura=
tion/ context before the readFields function is called?<br>

<br></div>This is the stack trace:<br><br>=A0=A0=A0=A0=A0=A0=A0 at org.apac=
he.hadoop.io.WritableComparator.compare(WritableComparator.java:103)<br>=A0=
=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.comp=
are(MapTask.java:1111)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.util.QuickSort.sortInternal(Quic=
kSort.java:70)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.util.QuickSort=
.sort(QuickSort.java:59)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapr=
ed.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.f=
lush(MapTask.java:1298)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapre=
d.MapTask$NewOutputCollector.close(MapTask.java:699)<br>=A0=A0=A0=A0=A0=A0=
=A0 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:=
370)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.Child$4.run(Child=
.java:255)<br>=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPri=
vileged(Native Method)<br>=A0=A0=A0=A0=A0=A0=A0 at javax.security.auth.Subj=
ect.doAs(Subject.java:415)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.security.UserGroupInformation.do=
As(UserGroupInformation.java:1149)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.h=
adoop.mapred.Child.main(Child.java:249)<br><br></div>
</blockquote></div><br></div>

--001a11334a3cc3788304e53464ea--