Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of chivas314159@gmail.com
 designates 209.85.192.178 as permitted sender)
MIME-Version: 1.0
Date: Sat, 31 Aug 2013 02:01:36 +0100
Message-ID: 
 <CAPZM1Te5sAKwso+bYR2z8hZc0Rg-KuOoidiO9+hgXCbOKrLO0w@mail.gmail.com>
Subject: Job config before read fields
From: Adrian CAPDEFIER <chivas314159@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b2e0d838a576c04e533e160

--047d7b2e0d838a576c04e533e160
Content-Type: text/plain; charset=ISO-8859-1

Howdy,

I apologise for the lack of code in this message, but the code is fairly
convoluted and it would obscure my problem. That being said, I can put
together some sample code if really needed.

I am trying to pass some metadata between the map & reduce steps. This
metadata is read and generated in the map step and stored in the job
config. It also needs to be recreated on the reduce node before the key/
value fields can be read in the readFields function.

I had assumed that I would be able to override the Reducer.setup() function
and that would be it, but apparently the readFields function is called
before the Reducer.setup() function.

My question is what is any (the best) place on the reduce node where I can
access the job configuration/ context before the readFields function is
called?

This is the stack trace:

        at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
        at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

--047d7b2e0d838a576c04e533e160
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Howdy,<br><br></div><div>I apologise for th=
e lack of code in this message, but the code is fairly convoluted and it wo=
uld obscure my problem. That being said, I can put together some sample cod=
e if really needed.<br>
</div><div><br></div>I am trying to pass some metadata between the map &amp=
; reduce steps. This metadata is read and generated in the map step and sto=
red in the job config. It also needs to be recreated on the reduce node bef=
ore the key/ value fields can be read in the readFields function.<br>
<br>I had assumed that I would be able to override the Reducer.setup() func=
tion and that would be it, but apparently the readFields function is called=
 before the Reducer.setup() function.<br><br></div>My question is what is a=
ny (the best) place on the reduce node where I can access the job configura=
tion/ context before the readFields function is called?<br>
<br></div>This is the stack trace:<br><br>=A0=A0=A0=A0=A0=A0=A0 at org.apac=
he.hadoop.io.WritableComparator.compare(WritableComparator.java:103)<br>=A0=
=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.comp=
are(MapTask.java:1111)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.util.QuickSort.sortInternal(Quic=
kSort.java:70)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.util.QuickSort=
.sort(QuickSort.java:59)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapr=
ed.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.f=
lush(MapTask.java:1298)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapre=
d.MapTask$NewOutputCollector.close(MapTask.java:699)<br>=A0=A0=A0=A0=A0=A0=
=A0 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:=
370)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.Child$4.run(Child=
.java:255)<br>=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPri=
vileged(Native Method)<br>=A0=A0=A0=A0=A0=A0=A0 at javax.security.auth.Subj=
ect.doAs(Subject.java:415)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.security.UserGroupInformation.do=
As(UserGroupInformation.java:1149)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.h=
adoop.mapred.Child.main(Child.java:249)<br><br></div>

--047d7b2e0d838a576c04e533e160--