hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew John <tmatthewjohn1...@gmail.com>
Subject Re: HDFS - MapReduce coupling
Date Mon, 02 May 2011 07:16:28 GMT
Any documentations on how the different daemons do the write/read on HDFS
and Local File System (direct), I mean the different protocols used in the
interactions. I basically wanted to figure out how intricate the coupling
between the Storage (HDFS + Local) and other processes in the Hadoop
infrastructure is.


On Mon, May 2, 2011 at 12:26 PM, Ted Dunning <tdunning@maprtech.com> wrote:

> Yes.  There is quite a bit of need for the local file system in clustered
> mode.
>
> For one think, all of the shuffle intermediate files are on local disk.
>  For
> another, the distributed cache is actually stored on local disk.
>
> HFDS is a frail vessel that cannot cope with all the needs.
>
> On Sun, May 1, 2011 at 11:48 PM, Matthew John <tmatthewjohn1988@gmail.com
> >wrote:
>
> > ...
> > 2) Does the Hadoop system utilize the local storage directly for any
> > purpose
> > (without going through the HDFS) in clustered mode?
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message