hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian CAPDEFIER <chivas314...@gmail.com>
Subject Job config before read fields
Date Sat, 31 Aug 2013 01:01:36 GMT
Howdy,

I apologise for the lack of code in this message, but the code is fairly
convoluted and it would obscure my problem. That being said, I can put
together some sample code if really needed.

I am trying to pass some metadata between the map & reduce steps. This
metadata is read and generated in the map step and stored in the job
config. It also needs to be recreated on the reduce node before the key/
value fields can be read in the readFields function.

I had assumed that I would be able to override the Reducer.setup() function
and that would be it, but apparently the readFields function is called
before the Reducer.setup() function.

My question is what is any (the best) place on the reduce node where I can
access the job configuration/ context before the readFields function is
called?

This is the stack trace:

        at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
        at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

Mime
View raw message