Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D4E2710D8D for ; Fri, 6 Jun 2014 05:53:38 +0000 (UTC) Received: (qmail 24581 invoked by uid 500); 6 Jun 2014 05:53:27 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 24442 invoked by uid 500); 6 Jun 2014 05:53:27 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 24435 invoked by uid 99); 6 Jun 2014 05:53:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Jun 2014 05:53:26 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rajkrrsingh@gmail.com designates 209.85.192.49 as permitted sender) Received: from [209.85.192.49] (HELO mail-qg0-f49.google.com) (209.85.192.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Jun 2014 05:53:23 +0000 Received: by mail-qg0-f49.google.com with SMTP id a108so3421023qge.36 for ; Thu, 05 Jun 2014 22:53:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=fcR12W77HP5wygwncPFYipjygwEJgHnMTpX7Exbjsy8=; b=oMj4bGkpSXoyQM56t4H2lp3MNZ5QSDBuU0mccH0dRoTYULnW89hksBAcbZE+pojmrV +FaQgu9zlxKg2rB7SVXKGwA5DROiBFE3htXQZeByHsOFE5eCnUGdwCSvDKTPhYtf9lXW svnSB7HkFZ3liRDuq5LZwkYd7wr5/DNCK7bO9pfsyv3ARoRKSl/LuP87nS0CbW8OMqcP ZdwVKtYkkZ0Wfg+w6KbdfiNHgjbaeYL6ufFWqbngaJdh1Zevn1aeU68ENPTsR2J7WZbJ oXidZhbJO0WnQkQnXhS7b341OKD+7UpD5PEcqvUAFioWg60jL+POnCtoVlbSC52f5S/Z AuIw== X-Received: by 10.140.30.70 with SMTP id c64mr4255018qgc.13.1402033982195; Thu, 05 Jun 2014 22:53:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.82.36 with HTTP; Thu, 5 Jun 2014 22:52:32 -0700 (PDT) In-Reply-To: References: From: Raj K Singh Date: Fri, 6 Jun 2014 11:22:32 +0530 Message-ID: Subject: Re: How can a task know if its running as a MR1 or MR2 job? To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11395d8478cbc504fb247926 X-Virus-Checked: Checked by ClamAV on apache.org --001a11395d8478cbc504fb247926 Content-Type: text/plain; charset=UTF-8 Hi I guess when we call submit() method that in tern call the setUseNewAPI() method to set the MR2 relevent setting.. here what i found after code walk through of hadoop private void setUseNewAPI() throws IOException { int numReduces = conf.getNumReduceTasks(); String oldMapperClass = "mapred.mapper.class"; String oldReduceClass = "mapred.reducer.class"; conf.setBooleanIfUnset("mapred.mapper.new-api", conf.get(oldMapperClass) == null); if(conf.getUseNewMapper()) { String mode = "new map API"; ensureNotSet("mapred.input.format.class", mode); ensureNotSet(oldMapperClass, mode); if(numReduces != 0) ensureNotSet("mapred.partitioner.class", mode); else ensureNotSet("mapred.output.format.class", mode); } else { String mode = "map compatability"; ensureNotSet("mapreduce.inputformat.class", mode); ensureNotSet("mapreduce.map.class", mode); if(numReduces != 0) ensureNotSet("mapreduce.partitioner.class", mode); else ensureNotSet("mapreduce.outputformat.class", mode); } if(numReduces != 0) { conf.setBooleanIfUnset("mapred.reducer.new-api", conf.get(oldReduceClass) == null); if(conf.getUseNewReducer()) { String mode = "new reduce API"; ensureNotSet("mapred.output.format.class", mode); ensureNotSet(oldReduceClass, mode); } else { String mode = "reduce compatability"; ensureNotSet("mapreduce.outputformat.class", mode); ensureNotSet("mapreduce.reduce.class", mode); } } } :::::::::::::::::::::::::::::::::::::::: Raj K Singh http://in.linkedin.com/in/rajkrrsingh http://www.rajkrrsingh.blogspot.com Mobile Tel: +91 (0)9899821370 On Tue, Jun 3, 2014 at 5:34 PM, Michael Segel wrote: > Just a quick question... > > Suppose you have a M/R job running. > How does the Mapper or Reducer task know or find out if its running as a > M/R 1 or M/R 2 job? > > I would suspect the job context would hold that information... but on > first glance I didn't see it. > So what am I missing? > > Thx > > -Mike > > > --001a11395d8478cbc504fb247926 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi=C2=A0

I guess when we call submit() method that in tern call the=C2=A0setUseNewAP= I() method to set the MR2 relevent setting.. here what i found after code w= alk through of hadoop

private void setUseNewAPI(= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 throws IOEx= ception
=C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 int numReduces =3D conf.getNumReduceTasks();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 String oldMapperClass =3D "mapred.= mapper.class";
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 String oldReduceClass =3D "mapred.reducer.class";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf.setBooleanIfU= nset("mapred.mapper.new-api", conf.get(oldMapperClass) =3D=3D nul= l);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if(conf.g= etUseNewMapper())
=C2=A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String mode =3D "new map API";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 e= nsureNotSet("mapred.input.format.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ensureNotSet(oldMapperClass, mode= );
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 if(numReduces !=3D 0)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ensureNotSet("mapred.partiti= oner.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 ensureNotSet("mapred.output.format.class", mode= );
=C2=A0 =C2=A0 =C2=A0 =C2=A0 } else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String mode =3D "map compatability&quo= t;;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 ensureNotSet("mapreduce.inputformat.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ensureNotSet("mapreduce.map.= class", mode);
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 if(numReduces !=3D 0)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ensureNotSet(&quo= t;mapreduce.partitioner.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 ensureNotSet("mapreduce.outputformat.class", mo= de);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
<= div class=3D"gmail_default"> =C2=A0 =C2=A0 =C2=A0 =C2=A0 if(numReduces !=3D 0)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 {
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 conf.setBooleanIfUnset("mapr= ed.reducer.new-api", conf.get(oldReduceClass) =3D=3D null);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if(conf.getUseNewReducer())
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 String mode =3D "new reduce API";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = ensureNotSet("mapred.output.format.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 ensureNotSet(oldReduceClass, mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 } else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = String mode =3D "reduce compatability";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 ensureNotSet("mapreduce.outputformat.class", mode);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 ensureNotSet("mapreduce.reduce.class", mode);
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 = =C2=A0 }

:::::::::::::::::= :::::::::::::::::::::::
Raj K Singh


On Tue, Jun 3, 2014 at 5:34 PM, Michael = Segel <msegel_hadoop@hotmail.com> wrote:
Just a quick question...

Suppose you have a M/R job running.
How does the Mapper or Reducer task know or find out if its running as a M/= R 1 or M/R 2 job?

I would suspect the job context would hold that information... but on first= glance I didn't see it.
So what am I missing?

Thx

-Mike



--001a11395d8478cbc504fb247926--