hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aastha Mehta <aasth...@gmail.com>
Subject Re: questions regarding fuse_dfs_read
Date Fri, 09 Sep 2011 19:36:52 GMT
Thanks Brian. That helped.

Regards,
Aastha.

On 7 September 2011 17:45, Brian Bockelman <bbockelm@cse.unl.edu> wrote:

> Hi Aastha,
>
> A read-ahead buffer is a common technique to trade higher bandwidth for
> lower latency for a number of common read patterns.  Your OS does something
> similar (a much more advanced technique though).  By reading ahead, HDFS is
> betting that your reads have a pattern to it.  I think the 10MB default is a
> touch excessive (made more sense in previous releases).  I use 32KB.
>
> The buffer is not used if you have very large reads, as it doesn't provide
> any benefit.
>
> Brian
>
> On Sep 7, 2011, at 12:45 AM, Aastha Mehta wrote:
>
> > Hello,
> >
> > I am using FUSE-DFS with HDFS for a project. I have to modify the read
> and
> > write functions of fuse_dfs. I have few questions regarding the
> > fuse_dfs_read code. There is an rdbuffer_size variable associated with
> the
> > dfs_context, which is by default initialized to 10M. What is this
> > rdbuffer_size and what is it used for?
> >
> > Secondly, in the fuse_dfs_read function, there are two places where
> > hdfsPread() is called in a loop. First, there is a check for whether the
> > requested read size is greater than the value of rdbuffer_size. Only if
> it
> > is, is the hdfsPread executed. In this case, the data is read into the
> > buffer passed from the caller.
> >
> > In the second case, hdfsPread is executed for if a valid buffer is
> > associated with the dfs file handle fh and the size and offset of read
> > request lie within the range of the fh->buf. In this case, the data is
> read
> > into fh->buf.
> >
> > Could someone explain what is happening here?
> >
> > Thanks,
> > Aastha.
> >
> > --
> > Aastha Mehta
> > B.E. (Hons.) Computer Science
> > BITS Pilani
> > E-mail: aasthakm@gmail.com
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message