Return-Path: 
 <mapreduce-user-return-2404-apmail-hadoop-mapreduce-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54BE147F7
	for <apmail-hadoop-mapreduce-user-archive@minotaur.apache.org>;
 Tue, 17 May 2011 10:47:41 +0000 (UTC)
Received: (qmail 50134 invoked by uid 500); 17 May 2011 10:47:40 -0000
Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org
Received: (qmail 50069 invoked by uid 500); 17 May 2011 10:47:40 -0000
Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:mapreduce-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:mapreduce-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:mapreduce-user@hadoop.apache.org>
List-Id: <mapreduce-user.hadoop.apache.org>
Reply-To: mapreduce-user@hadoop.apache.org
Delivered-To: mailing list mapreduce-user@hadoop.apache.org
Received: (qmail 50061 invoked by uid 99); 17 May 2011 10:47:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 10:47:40 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ygnhzeus@gmail.com designates
 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 10:47:35 +0000
Received: by qwj9 with SMTP id 9so221496qwj.35
        for <mapreduce-user@hadoop.apache.org>;
 Tue, 17 May 2011 03:47:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=8upBTVHXsFsA63+J0+Yk8sCn0I/k69bE86VqnF6203o=;
        b=t8IrLjGGP0EpXq7PlqmbiPBjaXGzKihBYfDNH9+ituIwVXm1xrnZq4MlEu5fJP4tkz
         hWqUa37JqbAQfVE9MLPlxkahOKjbkV+s5e8V+48/QRW/qEk1V+M9T+idn7+bURYkbPJX
         NJnrNl6vltFAj7ROLoHglZqikg2Uw7ojtB418=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=yBLtnmj+BBAaJ8+J8DpY+W/d25IVbRauePKMNPZKPAGszHO/lkSib8ttbz3mxuBL55
         98SxTcmLw5N2JQ6v284KVFG5FTTBOPhBl5GztqJEmlhr6pT9jnm5Sx96UROBTgO2BsCQ
         F/KPBwlvbm861I8QF3b1fu7H0gC12jgU1bimc=
MIME-Version: 1.0
Received: by 10.224.193.135 with SMTP id du7mr327553qab.213.1305629234069;
 Tue, 17 May 2011 03:47:14 -0700 (PDT)
Received: by 10.229.211.13 with HTTP; Tue, 17 May 2011 03:47:13 -0700 (PDT)
In-Reply-To: <BANLkTimHchJBvop_f6EtbisztJgf01f0QQ@mail.gmail.com>
References: <BANLkTimHchJBvop_f6EtbisztJgf01f0QQ@mail.gmail.com>
Date: Tue, 17 May 2011 18:47:13 +0800
Message-ID: <BANLkTimZPdBn3-TPR3Ung1RQTAMU2enohA@mail.gmail.com>
Subject: How to run or submit MapReduce Job to Hadoop in my own program?
From: =?GB2312?B?RmVsaXgu0Ow=?= <ygnhzeus@gmail.com>
To: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3005de96b4804704a3767e98

--20cf3005de96b4804704a3767e98
Content-Type: text/plain; charset=ISO-8859-1

Hi,all..How can I run a MR job though my own program instead of using
console to submit a job to a real Hadoop env?
I write code like this, this program works fine but i don't think it ran in
my Hadoop env,since nothing was produced in hadoop logs folder.
public int run(String[] args) throws Exception {

                Configuration config = new Configuration();
                config.set("fs.default.name", "hdfs://localhost:54310");
                config.set("mapred.job.tracker","localhost:54311");
                JobConf conf = new JobConf(config,CustomMR.class);
                conf.setJobName("custom aggregation");
                conf.setMapOutputKeyClass(Text.class);
                conf.setMapOutputValueClass(IntWritable.class);

            conf.setOutputFormat(HiveIgnoreKeyTextOutputFormat.class);

            conf.setMapperClass(MapClass.class);

            conf.setReducerClass(Reduce.class);
            TextInputFormat.setInputPaths(conf,args[0]);
            HiveIgnoreKeyTextOutputFormat.setOutputPath(conf, new
Path(args[1]));

            JobClient.runJob(conf);
                //conf
                return 0;
        }
public static void main(String[] args) throws Exception {
            args = new String[]{"/user/hive/warehouse/ptest","/testout"};
            int res = ToolRunner.run(new Configuration(), new CustomMR(),
args);
            System.exit(res);
}
Could some one give me an example?Thanks!

--20cf3005de96b4804704a3767e98
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote">Hi,all..How can I run a MR job though my own pro=
gram instead of using console to submit a job to a real Hadoop env?<br>I wr=
ite code like this, this program works fine but i don&#39;t think it ran in=
 my Hadoop env,since nothing was produced in hadoop logs folder.<br>

<span style=3D"color: rgb(51, 51, 255);">public int run(String[] args) thro=
ws Exception {</span><br style=3D"color: rgb(51, 51, 255);"><span style=3D"=
color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 </s=
pan><br style=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 Configuration config =3D new Configuration();</span><br style=
=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51, 255);">=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(&quot;</span><a style=
=3D"color: rgb(51, 51, 255);" href=3D"http://fs.default.name" target=3D"_bl=
ank">fs.default.name</a><span style=3D"color: rgb(51, 51, 255);">&quot;, &q=
uot;hdfs://localhost:54310&quot;);</span><br style=3D"color: rgb(51, 51, 25=
5);">
<span style=3D"color: rgb(51, 51, 255);">
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 config.set(&quot;mapred.job.<=
/span><span style=3D"color: rgb(51, 51, 255);">tracker&quot;,&quot;localhos=
t:54311&quot;);</span><br style=3D"color: rgb(51, 51, 255);"><span style=3D=
"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Jo=
bConf conf =3D new JobConf(config,CustomMR.class)</span><span style=3D"colo=
r: rgb(51, 51, 255);">;</span><br style=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 conf.setJobName(&quot;custom aggregation&quot;);</span><br sty=
le=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51, 255);">
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 conf.setMapOutputKeyClass(</s=
pan><span style=3D"color: rgb(51, 51, 255);">Text.class);</span><br style=
=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51, 255);">=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 conf.setMapOutputValueClass(</sp=
an><span style=3D"color: rgb(51, 51, 255);">IntWritable.class);</span><br s=
tyle=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 </span><br style=3D"color: rgb(51, 51, 255);"><span style=3D"c=
olor: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 conf.setOutputFo=
rmat(</span><span style=3D"color: rgb(51, 51, 255);">HiveIgnoreKeyTextOutpu=
tFormat.</span><span style=3D"color: rgb(51, 51, 255);">class);</span><br s=
tyle=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 </s=
pan><br style=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51=
, 255);">
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 conf.setMapperClass(MapClass.</span><span=
 style=3D"color: rgb(51, 51, 255);">class);=A0=A0=A0=A0=A0=A0=A0 </span><br=
 style=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51, 255);=
">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 </span><br style=3D"color: rgb(51, 51, 255=
);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =
conf.setReducerClass(Reduce.</span><span style=3D"color: rgb(51, 51, 255);"=
>class);</span><br style=3D"color: rgb(51, 51, 255);"><span style=3D"color:=
 rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 TextInputFormat.setIn=
putPaths(</span><span style=3D"color: rgb(51, 51, 255);">conf,args[0]);</sp=
an><br style=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =
HiveIgnoreKeyTextOutputFormat.</span><span style=3D"color: rgb(51, 51, 255)=
;">setOutputPath(conf, new Path(args[1]));</span><br style=3D"color: rgb(51=
, 51, 255);"><span style=3D"color: rgb(51, 51, 255);">
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 </span><br style=3D"color: rgb(51, 51, 255);=
"><span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0 JobClient.runJob(conf);</span><br style=3D"color: rgb(51, 51, 255);"><s=
pan style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0 //conf</span><br style=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 return 0;</span><br style=3D"color: rgb(51, 51, 255);"><span s=
tyle=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0 }</span><br style=
=3D"color: rgb(51, 51, 255);"><span style=3D"color: rgb(51, 51, 255);">publ=
ic static void main(String[] args) throws Exception {</span><br style=3D"co=
lor: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =
args =3D new String[]{&quot;/user/hive/</span><span style=3D"color: rgb(51,=
 51, 255);">warehouse/ptest&quot;,&quot;/testout&quot;};</span><br style=3D=
"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int res =3D ToolRunner.run(new Configurat=
ion(), new CustomMR(), args);</span><br style=3D"color: rgb(51, 51, 255);">=
<span style=3D"color: rgb(51, 51, 255);">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =
System.exit(res);</span><br style=3D"color: rgb(51, 51, 255);">
<span style=3D"color: rgb(51, 51, 255);">}</span><br>
</div>Could some one give me an example?Thanks!<br>

--20cf3005de96b4804704a3767e98--