hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavan Kulkarni <pavan.babu...@gmail.com>
Subject Doubt regarding use of HTTP during Shuffle phase
Date Wed, 27 Jun 2012 16:30:27 GMT
Hi,


 I am working on a project where I am trying to create hardlinks for the
Reducer nodes to the intermediate output files after Map phase during the
Shuffle phase
instead of  copying the files via HTTP which I assume would be more
efficient.
 I had few doubts regarding this. Why is HTTP used during the Shuffle phase
instead of just creating a Hardlink?
and Can you give pointers towards how do I create these hardlinks? I am
looking for where exactly can the intermediate Mapoutput Filename be
fetched?
Any help is highly appreciated .Thanks




-- 

--With Regards
Pavan Kulkarni

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message