hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谭军 <tanjun_2...@163.com>
Subject Re:Re: How do I set the intermediate output path when I use 2 mapreduce jobs?
Date Thu, 22 Sep 2011 00:27:32 GMT
Bobby Evans
Temp files are on HDFS not on local file system.



Jun Tan

At 2011-09-21 22:57:44,"Robert Evans" <evans@yahoo-inc.com> wrote:
Jun Tan,

So you want to have the temp file on the local file system, not on HDFS?  That is not going
to work, because there are other parts of the code that assume that they can see the file
(i.e. The splitter) which it cannot if it is only on the local file system of a remote host.
 It has to be stored in HDFS, or some other globally viewable file system.

--Bobby Evans

On 9/21/11 9:54 AM, "谭军" <tanjun_2525@163.com> wrote:

I want to use 2 MR jobs sequentially.
And the first job produces intermediate result to a temp file.
The second job reads the result in temp file but not the FileInputPath.
I tried, but FileNotFoundException reported.
Then I checked the datanodes, temp file was created.
The first job was executed correctly.
Why the second job cannot find the file? The file was created before the second job was executed.


Jun Tan

View raw message