Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.172 is neither permitted
 nor denied by domain of evans@yahoo-inc.com)
From: Robert Evans <evans@yahoo-inc.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Thu, 22 Sep 2011 10:08:12 -0700
Subject: Re: How do I set the intermediate output path when I use 2
 mapreduce jobs?
Thread-Topic: How do I set the intermediate output path when I use 2
 mapreduce jobs?
Thread-Index: Acx4vooS/t8BGEQxTvGfrjwQcghabQAi6t/L
Message-ID: <CAA0D5AC.297AC%evans@yahoo-inc.com>
In-Reply-To: <6d7aa922.f967.1328e87baa0.Coremail.tanjun_2525@163.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CAA0D5AC297ACevansyahooinccom_"
MIME-Version: 1.0

--_000_CAA0D5AC297ACevansyahooinccom_
Content-Type: text/plain; charset="gb2312"
Content-Transfer-Encoding: base64

U29ycnkgYWJvdXQgdGhlIGNvbmZ1c2lvbiB0aGVuLiAgTG9vayBhdCB0aGUgY29kZSB0aGF0IFN3
YXRoaSBzZW50LiAgWW91ciBwcm9ibGVtIGlzIHByb2JhYmx5IHdpdGggdGltaW5nIHNvbWUgd2hl
cmUuICBZb3UgbWF5IGJlIGxhdW5jaGluZyB0aGUgc2Vjb25kIGpvYiBiZWZvcmUgdGhlIGZpcnN0
IG9uZSBjb21wbGV0ZWx5IGZpbmlzaGVkLg0KDQotLUJvYmJ5IEV2YW5zDQoNCk9uIDkvMjEvMTEg
NzoyNyBQTSwgIsy3vvwiIDx0YW5qdW5fMjUyNUAxNjMuY29tPiB3cm90ZToNCg0KQm9iYnkgRXZh
bnMNClRlbXAgZmlsZXMgYXJlIG9uIEhERlMgbm90IG9uIGxvY2FsIGZpbGUgc3lzdGVtLg0KDQot
LQ0KUmVnYXJkcyENCg0KSnVuIFRhbg0KDQoNCkF0IDIwMTEtMDktMjEgMjI6NTc6NDQsIlJvYmVy
dCBFdmFucyIgPGV2YW5zQHlhaG9vLWluYy5jb20+IHdyb3RlOg0KSnVuIFRhbiwNCg0KU28geW91
IHdhbnQgdG8gaGF2ZSB0aGUgdGVtcCBmaWxlIG9uIHRoZSBsb2NhbCBmaWxlIHN5c3RlbSwgbm90
IG9uIEhERlM/ICBUaGF0IGlzIG5vdCBnb2luZyB0byB3b3JrLCBiZWNhdXNlIHRoZXJlIGFyZSBv
dGhlciBwYXJ0cyBvZiB0aGUgY29kZSB0aGF0IGFzc3VtZSB0aGF0IHRoZXkgY2FuIHNlZSB0aGUg
ZmlsZSAoaS5lLiBUaGUgc3BsaXR0ZXIpIHdoaWNoIGl0IGNhbm5vdCBpZiBpdCBpcyBvbmx5IG9u
IHRoZSBsb2NhbCBmaWxlIHN5c3RlbSBvZiBhIHJlbW90ZSBob3N0LiAgSXQgaGFzIHRvIGJlIHN0
b3JlZCBpbiBIREZTLCBvciBzb21lIG90aGVyIGdsb2JhbGx5IHZpZXdhYmxlIGZpbGUgc3lzdGVt
Lg0KDQotLUJvYmJ5IEV2YW5zDQoNCg0KT24gOS8yMS8xMSA5OjU0IEFNLCAizLe+/CIgPHRhbmp1
bl8yNTI1QDE2My5jb20gPGh0dHA6Ly9lZzVhODUubWFpbC4xNjMuY29tL2pzNC90YW5qdW5fMjUy
NUAxNjMuY29tPiA+IHdyb3RlOg0KDQpIaSwNCkkgd2FudCB0byB1c2UgMiBNUiBqb2JzIHNlcXVl
bnRpYWxseS4NCkFuZCB0aGUgZmlyc3Qgam9iIHByb2R1Y2VzIGludGVybWVkaWF0ZSByZXN1bHQg
dG8gYSB0ZW1wIGZpbGUuDQpUaGUgc2Vjb25kIGpvYiByZWFkcyB0aGUgcmVzdWx0IGluIHRlbXAg
ZmlsZSBidXQgbm90IHRoZSBGaWxlSW5wdXRQYXRoLg0KSSB0cmllZCwgYnV0IEZpbGVOb3RGb3Vu
ZEV4Y2VwdGlvbiByZXBvcnRlZC4NClRoZW4gSSBjaGVja2VkIHRoZSBkYXRhbm9kZXMsIHRlbXAg
ZmlsZSB3YXMgY3JlYXRlZC4NClRoZSBmaXJzdCBqb2Igd2FzIGV4ZWN1dGVkIGNvcnJlY3RseS4N
CldoeSB0aGUgc2Vjb25kIGpvYiBjYW5ub3QgZmluZCB0aGUgZmlsZT8gVGhlIGZpbGUgd2FzIGNy
ZWF0ZWQgYmVmb3JlIHRoZSBzZWNvbmQgam9iIHdhcyBleGVjdXRlZC4NClRoYW5rcyENCg0KLS0N
ClJlZ2FyZHMhDQoNCkp1biBUYW4NCg0KDQoNCg0KDQoNCg==

--_000_CAA0D5AC297ACevansyahooinccom_
Content-Type: text/html; charset="gb2312"
Content-Transfer-Encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>Re: How do I set the intermediate output path when I use 2 mapreduce=
 jobs?</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:=
11pt'>Sorry about the confusion then. &nbsp;Look at the code that Swathi se=
nt. &nbsp;Your problem is probably with timing some where. &nbsp;You may be=
 launching the second job before the first one completely finished.<BR>
<BR>
--Bobby Evans<BR>
<BR>
On 9/21/11 7:27 PM, &quot;=CC=B7=BE=FC&quot; &lt;<a href=3D"tanjun_2525@163=
.com">tanjun_2525@163.com</a>&gt; wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"=
><FONT SIZE=3D"4"><SPAN STYLE=3D'font-size:14pt'>Bobby Evans<BR>
Temp files are on HDFS not on local file system.<BR>
</SPAN></FONT></FONT><FONT FACE=3D"Arial"><SPAN STYLE=3D'font-size:10.5pt'>=
<BR>
--<BR>
Regards!<BR>
<BR>
Jun Tan<BR>
<BR>
<BR>
At 2011-09-21 22:57:44,&quot;Robert Evans&quot; &lt;<a href=3D"evans@yahoo-=
inc.com">evans@yahoo-inc.com</a>&gt; wrote:<BR>
</SPAN></FONT><BLOCKQUOTE><SPAN STYLE=3D'font-size:10.5pt'><FONT FACE=3D"Ca=
libri, Verdana, Helvetica, Arial">Jun Tan,<BR>
<BR>
So you want to have the temp file on the local file system, not on HDFS? &n=
bsp;That is not going to work, because there are other parts of the code th=
at assume that they can see the file (i.e. The splitter) which it cannot if=
 it is only on the local file system of a remote host. &nbsp;It has to be s=
tored in HDFS, or some other globally viewable file system.<BR>
<BR>
--Bobby Evans<BR>
<BR>
<BR>
On 9/21/11 9:54 AM, &quot;=CC=B7=BE=FC&quot; &lt;<a href=3D"tanjun_2525@163=
.com">tanjun_2525@163.com</a> &lt;<a href=3D"http://eg5a85.mail.163.com/js4=
/tanjun_2525@163.com">http://eg5a85.mail.163.com/js4/tanjun_2525@163.com</a=
>&gt; &gt; wrote:<BR>
<BR>
</FONT></SPAN><BLOCKQUOTE><SPAN STYLE=3D'font-size:10.5pt'><FONT FACE=3D"Ar=
ial">Hi,<BR>
I want to use 2 MR jobs sequentially.<BR>
And the first job produces intermediate result to a temp file.<BR>
The second job reads the result in temp file but not the FileInputPath.<BR>
I tried, but FileNotFoundException reported.<BR>
Then I checked the datanodes, temp file was created.<BR>
The first job was executed correctly.<BR>
Why the second job cannot find the file? The file was created before the se=
cond job was executed.<BR>
Thanks!<BR>
<BR>
--<BR>
Regards!<BR>
<BR>
Jun Tan<BR>
</FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
<BR>
</FONT></SPAN></BLOCKQUOTE></BLOCKQUOTE><SPAN STYLE=3D'font-size:10.5pt'><F=
ONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
<BR>
</FONT></SPAN></BLOCKQUOTE>
</BODY>
</HTML>


--_000_CAA0D5AC297ACevansyahooinccom_--