hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj K Singh <rajkrrsi...@gmail.com>
Subject Re: How can a task know if its running as a MR1 or MR2 job?
Date Fri, 06 Jun 2014 05:52:32 GMT
Hi

I guess when we call submit() method that in tern call the setUseNewAPI()
method to set the MR2 relevent setting.. here what i found after code walk
through of hadoop

private void setUseNewAPI()
        throws IOException
    {
        int numReduces = conf.getNumReduceTasks();
        String oldMapperClass = "mapred.mapper.class";
        String oldReduceClass = "mapred.reducer.class";
        conf.setBooleanIfUnset("mapred.mapper.new-api",
conf.get(oldMapperClass) == null);
        if(conf.getUseNewMapper())
        {
            String mode = "new map API";
            ensureNotSet("mapred.input.format.class", mode);
            ensureNotSet(oldMapperClass, mode);
            if(numReduces != 0)
                ensureNotSet("mapred.partitioner.class", mode);
            else
                ensureNotSet("mapred.output.format.class", mode);
        } else
        {
            String mode = "map compatability";
            ensureNotSet("mapreduce.inputformat.class", mode);
            ensureNotSet("mapreduce.map.class", mode);
            if(numReduces != 0)
                ensureNotSet("mapreduce.partitioner.class", mode);
            else
                ensureNotSet("mapreduce.outputformat.class", mode);
        }
        if(numReduces != 0)
        {
            conf.setBooleanIfUnset("mapred.reducer.new-api",
conf.get(oldReduceClass) == null);
            if(conf.getUseNewReducer())
            {
                String mode = "new reduce API";
                ensureNotSet("mapred.output.format.class", mode);
                ensureNotSet(oldReduceClass, mode);
            } else
            {
                String mode = "reduce compatability";
                ensureNotSet("mapreduce.outputformat.class", mode);
                ensureNotSet("mapreduce.reduce.class", mode);
            }
        }
    }

::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Tue, Jun 3, 2014 at 5:34 PM, Michael Segel <msegel_hadoop@hotmail.com>
wrote:

> Just a quick question...
>
> Suppose you have a M/R job running.
> How does the Mapper or Reducer task know or find out if its running as a
> M/R 1 or M/R 2 job?
>
> I would suspect the job context would hold that information... but on
> first glance I didn't see it.
> So what am I missing?
>
> Thx
>
> -Mike
>
>
>

Mime
View raw message