hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Public Network Services <publicnetworkservi...@gmail.com>
Subject Passing values from InputFormat via the Configuration object
Date Sat, 18 May 2013 00:33:21 GMT

I need to communicate some proprietary number (long) values from the
getSplits() method of a custom InputFormat class to the Hadoop driver class
(used to launch the job), but the JobContext object passed to the
getSplits() method has no access to a Counters object.

>From the source code, it seems that the Configuration object of the
launched job is passed around, so the JobContext object of getSplits() has
direct access to it via getConfiguration().

So, what about using a loop like

        Job job = ... // The launched job
        Configuration conf = job.getConfiguration();
        while (!job.isComplete()) {
        // Read the values from the configuration

from the driver class, which presumably runs in the same framework that
creates the splits?

The getSplits() method of the custom InputFormat would set each of the
values once.

All this does seem like a hack, so I would like some expert advice before
starting implementation. That is,

   1. Will it work?
   2. Is there a better method?


View raw message