hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <ang...@gmail.com>
Subject Re: why does not hdfs read ahead ?
Date Tue, 24 Nov 2009 18:35:03 GMT
Sequential read is the simplest case and it is pretty hard to improve upon
the current raw performance (HDFS client does take more CPU than one might
expect, Todd implemented an improvement for CPU consumed).

Just to reiterate what Todd said, there is an implicit read ahead for
sequential reads with TCP buffers and kernel read ahead on Datanodes.

If you extend the read ahead buffer to be more of a buffer cache for the
block, it could have big impact for some read access patterns (e.g. binary


On Mon, Nov 23, 2009 at 11:23 PM, Martin Mituzas <xietao1981@hotmail.com>wrote:

> I read the code and find the call
> DFSInputStream.read(buf, off, len)
> will cause the DataNode read len bytes (or less if encounting the end of
> block) , why does not hdfs read ahead to improve performance for sequential
> read?
> --
> View this message in context:
> http://old.nabble.com/why-does-not-hdfs-read-ahead---tp26491449p26491449.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message