Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=Jb5J8EQC3TPr1lTulDLD8HZgjzqfWJrf5ZfMa3VOZnYmMBtm///RwW4acGNF2vc/N9501hPhQMSIfzyxckj2g2txzrC8Ks0ROPmHDNYScbh9eBVzTsTFrTbx0WHTQ58gF3w6J+nd+x3XILgoleqG4uCsuJtRe1h+YbGirz6nbtc=;
Message-ID: <976356.41885.qm@web15908.mail.cnb.yahoo.com>
References: <8001.1258944821@buffalo.edu>
Date: Mon, 23 Nov 2009 13:33:35 +0800 (CST)
From: Gang Luo <lgpublic@yahoo.com.cn>
Subject: =?utf-8?B?UmXvvJogUmU6IFJlOiBSZe+8miBIZWxwIGluIEhhZG9vcA==?=
To: common-user@hadoop.apache.org, aa225@buffalo.edu
In-Reply-To: <8001.1258944821@buffalo.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

That is because it is not a file at all, but a directory. =0A=0AWhen you gi=
ve out the output path, it is actually a directory. The result file is name=
d part-00000 in that directory (it is default name, you can change it by so=
me settings). So, if you want to read it, you have to append the file name =
(that is 'part-00000' ) to the path, or just simply add a *.=0A=0Ayou can t=
ake a look at the example http://wiki.apache.org/hadoop/HadoopDfsReadWriteE=
xample=0A=0A. =0A=0ALuo, Gang=0A---------=0ADepartment of Computer Science=
=0ADuke University=0A(919)316-0993=0Agang.luo@duke.edu=0A=0A=0A=0A----- =E5=
=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6 ----=0A=E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=
=9A "aa225@buffalo.edu" <aa225@buffalo.edu>=0A=E6=94=B6=E4=BB=B6=E4=BA=BA=
=EF=BC=9A common-user@hadoop.apache.org; aa225@buffalo.edu; Jason Venner <j=
ason.hadoop@gmail.com>=0A=E5=8F=91=E9=80=81=E6=97=A5=E6=9C=9F=EF=BC=9A 2009=
/11/22 (=E5=91=A8=E6=97=A5) 9:53:41 =E4=B8=8B=E5=8D=88=0A=E4=B8=BB   =E9=A2=
=98=EF=BC=9A Re: Re: Re: Re=EF=BC=9A Help in Hadoop=0A=0AI am still getting=
 the same exception. This is the stack trace of it.=0A=0Ajava.io.IOExceptio=
n: Not a file: hdfs://zeus:18004/user/hadoop/output6/MatrixA-Row1=0A       =
 at=0Aorg.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.ja=
va:195)=0A        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient=
.java:797)=0A        at org.apache.hadoop.mapred.JobClient.runJob(JobClient=
.java:1142)=0A        at MatrixMultiplication.main(MatrixMultiplication.jav=
a:229)=0A        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Met=
hod)=0A        at=0Asun.reflect.NativeMethodAccessorImpl.invoke(NativeMetho=
dAccessorImpl.java:39)=0A        at=0Asun.reflect.DelegatingMethodAccessorI=
mpl.invoke(DelegatingMethodAccessorImpl.java:25)=0A        at java.lang.ref=
lect.Method.invoke(Method.java:597)=0A        at org.apache.hadoop.util.Run=
Jar.main(RunJar.java:165)=0A        at org.apache.hadoop.mapred.JobShell.ru=
n(JobShell.java:54)=0A        at org.apache.hadoop.util.ToolRunner.run(Tool=
Runner.java:65)=0A        at org.apache.hadoop.util.ToolRunner.run(ToolRunn=
er.java:79)=0A        at org.apache.hadoop.mapred.JobShell.main(JobShell.ja=
va:68)=0A=0A=0AAbhishek Agrawal=0A=0ASUNY- Buffalo=0A(716-435-7122)=0A=0AOn=
 Sun 11/22/09  9:28 PM , Jason Venner jason.hadoop@gmail.com sent:=0A> set =
the number of reduce tasks to 1.=0A> =0A> 2009/11/22  =0A> Hi everybody,=0A=
> =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD The 10 differ=
ent map-reducers store their=0A> respective outputs in 10=0A> different fil=
es. This is the snap shot=0A> hadoop@zeus:~/hadoop-0.19.1$ bin/hadoop dfs -=
ls output5=0A> Found 2 items=0A> drwxr-xr-x =E8=81=BD - hadoop supergroup =
=E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD0 2003-05-16 02:16=0A> /us=
er/hadoop/output5/MatrixA-Row1=0A> drwxr-xr-x =E8=81=BD - hadoop supergroup=
 =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD =E8=81=BD0 2003-05-16 02:16=0A> /u=
ser/hadoop/output5/MatrixA-Row2=0A> Now when I try to open any of these fil=
es I get an error message=0A> hadoop@zeus:~/hadoop-0.19.1$ bin/hadoop dfs -=
cat=0A> output5/MatrixA-Row1=0A> cat: Source must be a file.=0A> hadoop@zeu=
s:~/hadoop-0.19.1$=0A> But if I run=0A> hadoop@zeus:~/hadoop-0.19.1$ bin/ha=
doop dfs -cat=0A> output5/MatrixA-Row1/part-00000=0A> I get the correct out=
put. I do not understand why I have to give=0A> this extra=0A> "part-00000"=
. Now when I run a map reduce task to merge the outputs=0A> of all the=0A> =
files, I give the name of the directory output5 as the Input path.=0A> But =
I get a=0A> bug saying=0A> java.io.IOException: Not a file:=0A> hdfs://zeus=
:18004/user/hadoop/output5/MatrixA-Row1=0A> I cannot understand how to make=
 the frame work read my files.=0A> Alternatively I tried to avoid the map r=
educe approach for combining=0A> files and do=0A> it via a simple program, =
but I am unable to start. Can some one give=0A> me some=0A> sample implemen=
tation or something.=0A> Any help is appreciated=0A> Thank You=0A> Abhishek=
 Agrawal=0A> SUNY- Buffalo=0A> (716-435-7122)=0A> On Sun 11/22/09 =E8=81=BD=
5:48 PM ,  sent:=0A> > Hellow,=0A> > If I write the output of the 10 tasks =
in 10 different files then=0A> how do=0A> > Igo about merging the output ? =
Is there some in built=0A> functionality or do I=0A> > haveto write some co=
de for that ?=0A> >=0A> > Thank You=0A> >=0A> > Abhishek Agrawal=0A> >=0A> =
> SUNY- Buffalo=0A> > (716-435-7122)=0A> >=0A> > On Sun 11/22/09 =E8=81=BD5=
:40 PM , Gang Luo lgpubli=0A> >  sent:> Hi. If the output path already exis=
ts, it seems=0A> > you could not execute any> task with the same output pat=
h. I think=0A> you can=0A> > output the results of the> 10 tasks to 10 diff=
erent paths, and=0A> then do sth=0A> > more (by the 11th task, for> example=
) to merge the 10 results into=0A> 1 file.=0A> > >=0A> > > Gang Luo=0A> > >=
 ---------=0A> > > Department of Computer Science=0A> > > Duke University=
=0A> > > (919)316-0993=0A> > > gang.luo@du> ke.edu [4]=0A> > >=0A> > >=0A> =
> > -----=0A> > =E8=84=99=E6=A5=BC=E8=8C=82=E9=A9=B4=E9=99=86=E8=8C=82=E9=
=A9=B4=E9=99=86=E8=84=99=E6=A5=BC&Acir=0A> > c;=E6=90=82=E8=8C=82=E9=A9=B4=
=E9=99=86=E8=84=99=E6=BC=8F=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=97&r=0A> > eg;=
=E8=84=99=E9=99=87=E8=84=97=E7=A6=84=E8=84=97=E9=9C=B2 ---->=0A> > =E8=84=
=99=E6=A5=BC=E8=8C=82=E9=A9=B4=E9=99=86=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=
=E9=99=87&A=0A> > circ;=E7=A6=84=E8=84=97=E9=9C=B2=E8=84=99=E9=99=87=E8=84=
=97=E6=BD=9E=E8=84=97=E6=BD=9E=E8=84=99=0A> > =E7=82=89=E8=84=97=E5=BD=95=
=E8=8C=82=E9=A9=B4=E9=99=86 "aa225@buffa> lo.edu [5]" =0A> > lo.edu [6]>=E8=
=84=99=E5=A8=84=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=97=E9=9C=B2=E8=84=99&curre=
=0A> > n;=E8=84=97=E7=A6=84=E8=84=97=E9=9C=B2=E8=84=99=E9=99=87=E8=84=97=E6=
=BD=9E=E8=84=97=E6=BD=9E&Ati=0A> > lde;=E7=82=89=E8=84=97=E5=BD=95=E8=8C=82=
=E9=A9=B4=E9=99=86 common-user@hadoop.apache.org=E8=84=99=E6=A5=BC=E8=8C=82=
=E9=A9=B4=E9=99=86=E8=8C=82=0A> > =E9=A9=B4=E9=99=86=E8=84=99=E6=BC=8F=E8=
=8C=82=E9=A9=B4=E9=99=86=E8=8C=82=E9=A9=B4=E9=99=86&=0A> > Atilde;=E5=A8=84=
=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=97=E6=A5=BC=E8=84=99=E5=A8=84=E8=8C=82&iq=
u=0A> > est;=E9=99=86=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E7=82=89=E8=84=97=
=E5=BD=95=E8=8C=82&iques=0A> > t;=E9=99=86 2009/11/22>=0A> > (=E8=84=99=E6=
=A5=BC=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=97=E7=AF=93=E8=84=99=E5=A8=84=E8=8C=
=82&ique=0A> > st;=E9=99=86=E8=84=97=E6=A5=BC) 5:25:55=0A> > =E8=84=99=E9=
=99=87=E8=84=97=E8=B5=82=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E6=A5=BC=E8=8C=
=82&iqu=0A> > est;=E9=99=86=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E9=99=87=E8=
=84=97=E8=B5=82=E8=84=97&raq=0A> > uo;=0A> > =E8=84=99=E6=BC=8F=E8=84=97=E5=
=9E=84=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E7=82=89=E8=84=97&frac=0A> > 14;=
=E8=8C=82=E9=A9=B4=E9=99=86 Help in Hadoop>=0A> > > Hello Everybody,=0A> > =
> I have a doubt in a map reduce program and I=0A> > would appreciate any> =
help. I run the program using the command=0A> > bin/hadoop jar HomeWork.jar=
 prg1> inputoutput. Ideally from within=0A> prg1, I want to=0A> > sequentia=
lly launch 10 map-> reducetasks. I want to store the=0A> output of all=0A> =
> these map reduce tasks in some> file.Currently I have kept the=0A> input =
format and=0A> > output format of the jobs as> TextInputFormat and TextOutp=
utFormat=0A> > respectively. Now I have the> followingquestions.=0A> > >=0A=
> > > 1. When I run more than 1 task from the same=0A> > program, the outpu=
t file of> all thetasks is same. The framework=0A> does not=0A> > allows th=
e 2 =E8=81=BD map reduce task to> have thesame output file as=0A> task 1.=
=0A> > >=0A> > > 2. Before the 2 task launches I also get this=0A> > error =
>=0A> > > Cannot initialize JVM Metrics with=0A> > processName=3DJobTracker=
, sessionId=3D -> alreadyinitialized=0A> > >=0A> > > 3. When the 2 map redu=
ce tasks writes its output=0A> > to file> "output", wont theprevious conten=
t of=0A> > this file get over written ?>=0A> > > Thank You=0A> > >=0A> > > =
Abhishek Agrawal=0A> > >=0A> > > SUNY- Buffalo=0A> > > (716-435-7122)=0A> >=
 >=0A> > >=0A> > >=0A> > __________________________________________________=
_________ >=0A> > =E8=84=99=E6=A5=BC=E8=84=97=E6=A5=BC=E8=84=97=E9=99=86=E8=
=84=99=E6=90=82=E8=8C=82=E9=A9=B4=E9=99=86=0A> > ;=E8=84=97=E6=BC=8F=E8=84=
=99=E7=AF=93=E8=84=97=E9=BA=93=E8=84=97=E6=BD=9E=E8=84=99=E6=A5=BC=E8=8C=82=
&i=0A> > quest;=E9=99=86=E8=84=97=E9=9A=86&cce>=0A> > dil;=E8=84=97=E9=A2=
=85=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E9=99=87=E8=84=97=E9=99=86=E8=84=97=
=0A> > ;=E8=81=BD=E8=84=99=E6=A5=BC=E8=8C=82=E9=A9=B4=E9=99=86=E8=8C=82=E9=
=A9=B4=E9=99=86=E8=84=99&ma=0A> > cr;=E8=84=97=E5=BD=95=E8=8C=82=E9=A9=B4=
=E9=99=86=E8=84=99=E6=BC=8F=E8=8C=82=E9=A9=B4=E9=99=86=0A> > ;=E8=84=97=E5=
=BA=90=E8=84=99=E6=90=82>=0A> > ;=E8=84=97=E5=BA=90=E8=84=97=E5=8D=A4=E8=84=
=99=E7=AF=93=E8=84=97=E9=BA=93=E8=84=97=E6=BD=9E=E8=84=99=0A> > ;=E6=A5=BC=
=E8=8C=82=E9=A9=B4=E9=99=86=E8=84=97=E9=9A=86=E8=84=99=E6=A5=BC=E8=8C=82=E9=
=A9=B4&frac1=0A> > 2;=E8=84=97=E7=AF=93=E8=84=99=E5=A8=84=E8=8C=82=E9=A9=B4=
=E9=99=86=E8=84=97=E6=8E=B3>=0A> > ;=E8=84=99=E9=99=87=E8=84=97=E8=B5=82=E8=
=8C=82=E9=A9=B4=E9=99=86=E8=84=99=E6=90=82=E8=84=97&=0A> > ordm;=E8=84=97=
=E9=A9=B4=E8=84=99=E7=82=89=E8=84=97=E5=BD=95=E8=8C=82=E9=A9=B4=E9=99=86 ht=
tp://card.mail.cn.yahoo.com/ [7]>=0A> > >=0A> > >=0A> >=0A> >=0A> >=0A> -- =
=0A> Pro Hadoop, a book to guide you from beginner to hadoop mastery,=0A> h=
ttp://www.amazon.com/dp/1430219424?tag=3Djewlerymall [8]=0A> www.prohadoopb=
ook.com [9] a community for Hadoop Professionals=0A> =0A> =0A> Links:=0A> -=
-----=0A> [4] http://ke.edu=0A> [5] http://lo.edu=0A> [6] http://lo.edu=0A>=
 [7] http://card.mail.cn.yahoo.com/=0A> [8] http://www.amazon.com/dp/143021=
9424?tag=3Djewlerymall=0A> [9] http://www.prohadoopbook.com=0A> =0A> =0A=0A=
=0A      ___________________________________________________________ =0A  =
=E5=A5=BD=E7=8E=A9=E8=B4=BA=E5=8D=A1=E7=AD=89=E4=BD=A0=E5=8F=91=EF=BC=8C=E9=
=82=AE=E7=AE=B1=E8=B4=BA=E5=8D=A1=E5=85=A8=E6=96=B0=E4=B8=8A=E7=BA=BF=EF=BC=
=81 =0Ahttp://card.mail.cn.yahoo.com/