hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <cdoug...@apache.org>
Subject Re: Sync-marker in uncompressed sequenceFile
Date Mon, 21 Mar 2011 20:07:26 GMT
It's used to align input splits of the SequenceFile. A reader can
start at an arbitrary offset, then find the boundary of the next block
of records by looking for the sync marker defined in the header. -C

On Mon, Mar 21, 2011 at 7:40 AM, Weishung Chung <weishung@gmail.com> wrote:
> Hello my fellow Hadoop users/developers,
> I'm reading the SequenceFile source code, and there is a checkAndWriteSync()
> method that writes a sync marker every so many bytes. I was wondering what's
> the use of the sync marker. I know one can use it to designate the end of a
> header, but it's also used in the process of writing the uncompressed data.
>  I wish I could have figured it out :(
> Thank you so much

View raw message