hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavan Kulkarni <pavan.babu...@gmail.com>
Subject Re: Doubt regarding use of HTTP during Shuffle phase
Date Wed, 27 Jun 2012 16:44:04 GMT
Yes you are correct, but we can use Lustre FS and it does scale right?
I am new to this so please excuse if I am wrong in some assumptions.Thanks

On Wed, Jun 27, 2012 at 9:40 AM, Owen O'Malley <omalley@apache.org> wrote:

> On Wed, Jun 27, 2012 at 9:33 AM, Pavan Kulkarni <pavan.baburao@gmail.com
> >wrote:
>
> Why is HTTP used during the Shuffle phase
> > instead of just creating a Hardlink?
> >
>
> The reduces run on a separate machine that the map. Hadoop doesn't use a
> shared NFS file system between the machines, because it doesn't scale.
>
> -- Owen
>



-- 

--With Regards
Pavan Kulkarni

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message