hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: Re: Re: Re: Help in Hadoop
Date Mon, 23 Nov 2009 05:33:35 GMT
That is because it is not a file at all, but a directory. 

When you give out the output path, it is actually a directory. The result file is named part-00000
in that directory (it is default name, you can change it by some settings). So, if you want
to read it, you have to append the file name (that is 'part-00000' ) to the path, or just
simply add a *.

you can take a look at the example http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample

. 

Luo, Gang
---------
Department of Computer Science
Duke University
(919)316-0993
gang.luo@duke.edu



----- 原始邮件 ----
发件人: "aa225@buffalo.edu" <aa225@buffalo.edu>
收件人: common-user@hadoop.apache.org; aa225@buffalo.edu; Jason Venner <jason.hadoop@gmail.com>
发送日期: 2009/11/22 (周日) 9:53:41 下午
主   题: Re: Re: Re: Re: Help in Hadoop

I am still getting the same exception. This is the stack trace of it.

java.io.IOException: Not a file: hdfs://zeus:18004/user/hadoop/output6/MatrixA-Row1
        at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:195)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
        at MatrixMultiplication.main(MatrixMultiplication.java:229)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)


Abhishek Agrawal

SUNY- Buffalo
(716-435-7122)

On Sun 11/22/09  9:28 PM , Jason Venner jason.hadoop@gmail.com sent:
> set the number of reduce tasks to 1.
> 
> 2009/11/22  
> Hi everybody,
> 聽 聽 聽 聽 聽 聽 The 10 different map-reducers store their
> respective outputs in 10
> different files. This is the snap shot
> hadoop@zeus:~/hadoop-0.19.1$ bin/hadoop dfs -ls output5
> Found 2 items
> drwxr-xr-x 聽 - hadoop supergroup 聽 聽 聽 聽 聽0 2003-05-16 02:16
> /user/hadoop/output5/MatrixA-Row1
> drwxr-xr-x 聽 - hadoop supergroup 聽 聽 聽 聽 聽0 2003-05-16 02:16
> /user/hadoop/output5/MatrixA-Row2
> Now when I try to open any of these files I get an error message
> hadoop@zeus:~/hadoop-0.19.1$ bin/hadoop dfs -cat
> output5/MatrixA-Row1
> cat: Source must be a file.
> hadoop@zeus:~/hadoop-0.19.1$
> But if I run
> hadoop@zeus:~/hadoop-0.19.1$ bin/hadoop dfs -cat
> output5/MatrixA-Row1/part-00000
> I get the correct output. I do not understand why I have to give
> this extra
> "part-00000". Now when I run a map reduce task to merge the outputs
> of all the
> files, I give the name of the directory output5 as the Input path.
> But I get a
> bug saying
> java.io.IOException: Not a file:
> hdfs://zeus:18004/user/hadoop/output5/MatrixA-Row1
> I cannot understand how to make the frame work read my files.
> Alternatively I tried to avoid the map reduce approach for combining
> files and do
> it via a simple program, but I am unable to start. Can some one give
> me some
> sample implementation or something.
> Any help is appreciated
> Thank You
> Abhishek Agrawal
> SUNY- Buffalo
> (716-435-7122)
> On Sun 11/22/09 聽5:48 PM ,  sent:
> > Hellow,
> > If I write the output of the 10 tasks in 10 different files then
> how do
> > Igo about merging the output ? Is there some in built
> functionality or do I
> > haveto write some code for that ?
> >
> > Thank You
> >
> > Abhishek Agrawal
> >
> > SUNY- Buffalo
> > (716-435-7122)
> >
> > On Sun 11/22/09 聽5:40 PM , Gang Luo lgpubli
> >  sent:> Hi. If the output path already exists, it seems
> > you could not execute any> task with the same output path. I think
> you can
> > output the results of the> 10 tasks to 10 different paths, and
> then do sth
> > more (by the 11th task, for> example) to merge the 10 results into
> 1 file.
> > >
> > > Gang Luo
> > > ---------
> > > Department of Computer Science
> > > Duke University
> > > (919)316-0993
> > > gang.luo@du> ke.edu [4]
> > >
> > >
> > > -----
> > 脙楼茂驴陆茂驴陆脙楼&Acir
> > c;搂茂驴陆脙漏茂驴陆脗&r
> > eg;脙陇脗禄脗露 ---->
> > 脙楼茂驴陆茂驴陆脙陇&A
> > circ;禄脗露脙陇脗潞脗潞脙
> > 炉脗录茂驴陆 "aa225@buffa> lo.edu [5]" 
> > lo.edu [6]>脙娄茂驴陆脗露脙&curre
> > n;脗禄脗露脙陇脗潞脗潞&Ati
> > lde;炉脗录茂驴陆 common-user@hadoop.apache.org脙楼茂驴陆茂
> > 驴陆脙漏茂驴陆茂驴陆&
> > Atilde;娄茂驴陆脗楼脙娄茂&iqu
> > est;陆茂驴陆脙炉脗录茂&iques
> > t;陆 2009/11/22>
> > (脙楼茂驴陆脗篓脙娄茂&ique
> > st;陆脗楼) 5:25:55
> > 脙陇脗赂茂驴陆脙楼茂&iqu
> > est;陆茂驴陆脙陇脗赂脗&raq
> > uo;
> > 脙漏脗垄茂驴陆脙炉脗&frac
> > 14;茂驴陆 Help in Hadoop>
> > > Hello Everybody,
> > > I have a doubt in a map reduce program and I
> > would appreciate any> help. I run the program using the command
> > bin/hadoop jar HomeWork.jar prg1> inputoutput. Ideally from within
> prg1, I want to
> > sequentially launch 10 map-> reducetasks. I want to store the
> output of all
> > these map reduce tasks in some> file.Currently I have kept the
> input format and
> > output format of the jobs as> TextInputFormat and TextOutputFormat
> > respectively. Now I have the> followingquestions.
> > >
> > > 1. When I run more than 1 task from the same
> > program, the output file of> all thetasks is same. The framework
> does not
> > allows the 2 聽 map reduce task to> have thesame output file as
> task 1.
> > >
> > > 2. Before the 2 task launches I also get this
> > error >
> > > Cannot initialize JVM Metrics with
> > processName=JobTracker, sessionId= -> alreadyinitialized
> > >
> > > 3. When the 2 map reduce tasks writes its output
> > to file> "output", wont theprevious content of
> > this file get over written ?>
> > > Thank You
> > >
> > > Abhishek Agrawal
> > >
> > > SUNY- Buffalo
> > > (716-435-7122)
> > >
> > >
> > >
> > ___________________________________________________________ >
> > 脙楼脗楼脗陆脙搂茂驴陆
> > ;脗漏脙篓脗麓脗潞脙楼茂&i
> > quest;陆脗隆&cce>
> > dil;脗颅茂驴陆脙陇脗陆脗
> > ;聽脙楼茂驴陆茂驴陆脙&ma
> > cr;脗录茂驴陆脙漏茂驴陆
> > ;脗庐脙搂>
> > ;脗庐脗卤脙篓脗麓脗潞脙
> > ;楼茂驴陆脗隆脙楼茂驴&frac1
> > 2;脗篓脙娄茂驴陆脗掳>
> > ;脙陇脗赂茂驴陆脙搂脗&
> > ordm;脗驴脙炉脗录茂驴陆 http://card.mail.cn.yahoo.com/ [7]>
> > >
> > >
> >
> >
> >
> -- 
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall [8]
> www.prohadoopbook.com [9] a community for Hadoop Professionals
> 
> 
> Links:
> ------
> [4] http://ke.edu
> [5] http://lo.edu
> [6] http://lo.edu
> [7] http://card.mail.cn.yahoo.com/
> [8] http://www.amazon.com/dp/1430219424?tag=jewlerymall
> [9] http://www.prohadoopbook.com
> 
> 


      ___________________________________________________________ 
  好玩贺卡等你发,邮箱贺卡全新上线! 
http://card.mail.cn.yahoo.com/

Mime
View raw message