hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: How do I set the intermediate output path when I use 2 mapreduce jobs?
Date Wed, 21 Sep 2011 14:57:44 GMT
Jun Tan,

So you want to have the temp file on the local file system, not on HDFS?  That is not going
to work, because there are other parts of the code that assume that they can see the file
(i.e. The splitter) which it cannot if it is only on the local file system of a remote host.
 It has to be stored in HDFS, or some other globally viewable file system.

--Bobby Evans

On 9/21/11 9:54 AM, "谭军" <tanjun_2525@163.com> wrote:

I want to use 2 MR jobs sequentially.
And the first job produces intermediate result to a temp file.
The second job reads the result in temp file but not the FileInputPath.
I tried, but FileNotFoundException reported.
Then I checked the datanodes, temp file was created.
The first job was executed correctly.
Why the second job cannot find the file? The file was created before the second job was executed.


Jun Tan

View raw message