hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuanyuan Tian <yt...@us.ibm.com>
Subject Re: conf.get("map.input.file") returns null when using MultipleInputs in Hadoop 0.20
Date Sun, 02 May 2010 22:49:49 GMT

Hi Farhan,

I believe I have to use the old JobConf MapReduce interface in order to
user MultipleInputs. As a result, I cannot do as you suggested.

Yuanyuan


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Farhan Husain <farhan.husain@csebuet.org>                                       
                                                                 |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |common-user@hadoop.apache.org                                                         
                                                           |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |04/30/2010 11:46 PM                                                                   
                                                           |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: conf.get("map.input.file") returns null when using MultipleInputs 	in Hadoop 0.20 
                                                          |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Can you try the following code?

((FileSplit) context.getInputSplit()).getPath().getName()

Thanks,
Farhan

On Thu, Apr 29, 2010 at 12:46 PM, Tom White <tom@cloudera.com> wrote:

> Hi Yuanyuan,
>
> Thanks for filing an issue. To work around the issue could you use a
> regular FileInputFormat in a set of map-only jobs (which can read the
> input file names) so you can create a common input for a final MR job?
> This is admittedly less efficient since it needs more jobs.
>
> Cheers,
> Tom
>
> On Thu, Apr 29, 2010 at 10:37 AM, Yuanyuan Tian <ytian@us.ibm.com> wrote:
> >
> > Hi Tom,
> >
> > I have file a JIRA ticket (MAPREDUCE-1743) for this issue. At the mean
> time, can you suggest an alternative approach to achieve what I want
> (supporting different input formats and get the input file name in each
> mapper)?
> >
> > Yuanyuan
> >
> > Tom White ---04/29/2010 09:42:44 AM---Hi Yuanyuan, I think you've found
a
> bug - could you file a JIRA issue for this please?
> >
> >
> > From:
> > Tom White <tom@cloudera.com>
> > To:
> > common-user@hadoop.apache.org
> > Date:
> > 04/29/2010 09:42 AM
> > Subject:
> > Re: conf.get("map.input.file") returns null when using MultipleInputs
in
> Hadoop 0.20
> > ________________________________
> >
> >
> > Hi Yuanyuan,
> >
> > I think you've found a bug - could you file a JIRA issue for this
please?
> >
> > Thanks,
> > Tom
> >
> > On Wed, Apr 28, 2010 at 11:04 PM, Yuanyuan Tian <ytian@us.ibm.com>
> wrote:
> > >
> > >
> > > I have a problem in getting the input file name in the mapper  when
> uisng
> > > MultipleInputs. I need to use MultipleInputs to support different
> formats
> > > for my inputs to the my MapReduce job. And inside each mapper, I also
> need
> > > to know the exact input file that the mapper is processing. However,
> > > conf.get("map.input.file") returns null. Can anybody help me solve
this
> > > problem? Thanks in advance.
> > >
> > > public class Test extends Configured implements Tool{
> > >
> > >        static class InnerMapper extends MapReduceBase implements
> > > Mapper<Writable, Writable, NullWritable, Text>
> > >        {
> > >                ................
> > >                ................
> > >
> > >                public void configure(JobConf conf)
> > >                {
> > >                        String inputName=conf.get("map.input.file"));
> > >                        .......................................
> > >                }
> > >
> > >        }
> > >
> > >        public int run(String[] arg0) throws Exception {
> > >                JonConf job;
> > >                job = new JobConf(Test.class);
> > >                ...........................................
> > >
> > >                MultipleInputs.addInputPath(conf, new Path("A"),
> > > TextInputFormat.class);
> > >                MultipleInputs.addInputPath(conf, new Path("B"),
> > > SequenceFileFormat.class);
> > >                ...........................................
> > >        }
> > > }
> > >
> > > Yuanyuan
> >
> >
>


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message