hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Reading Records from a Sequence File
Date Fri, 01 Apr 2011 06:59:34 GMT
On Fri, Apr 1, 2011 at 9:00 AM, maha <maha@umail.ucsb.edu> wrote:
> Hello Everyone,
>        As far as I know, when my java program opens a sequence file for a map calculations,
from hdfs. Using SequenceFile.Reader(key,value) will actually read the file in dfs.block.size
then grabes record-by-record from memory.
>  Is that right?

The dfs.block.size part is partially right when applied in MapReduce
(actually, it would look for sync points for read start and read end).
And no, the reader does not load the entire data in the memory in
one-go. It buffers and reads off the stream just like any other

Could we have some more information on what your java program does,
and what exactly you are measuring? :)

Harsh J

View raw message