hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Sripati <praveensrip...@gmail.com>
Subject Re: SequenceFile with one very large value
Date Mon, 05 Dec 2011 15:46:49 GMT
>> SequenceFiles place sync markers (similar to what 'newlines' mean in
text files) after  a bunch of records, and that is the reason why your
record does not split when read.

Sync is placed after every N records and is used for moving from an
arbitrary location in a file to a start of the next record. A mapper
processes a sequence file block from the first sync in the current block to
the first sync in the next block in the file. There might be some data
transfer from other node to the node where the task is running.

What happens with a block size of 128MB, a key more than 128 MB and a
particular block doesn't have a sync mark? Will the mapper see that there
is no sync mark in the block and doesn't do anything or the block is not
assigned to a mapper?

Regards,
Praveen

On Mon, Dec 5, 2011 at 10:47 AM, Harsh J <harsh@cloudera.com> wrote:

> Florin,
>
> Based on the SequenceFileInputFormat's splitting, you should see just
> one task reading the record. SequenceFiles place sync markers (similar
> to what 'newlines' mean in text files) after  a bunch of records, and
> that is the reason why your record does not split when read.
>
> Also worth thinking about increasing block size for these files to fit
> their contents.
>
> On Thu, Oct 27, 2011 at 9:31 PM, Florin P <florinpico@yahoo.com> wrote:
> > Hello!
> >  Suppose this scenario:
> > 1. The DFS block 64MB
> > 2. We populate a SequenceFile with a binary value that has 200MB (that
> represents a PDF file)
> > In the circumstances of above scenario:
> > 1. How many blocks will be created on HDFS?
> > 2. The number of blocks will be 200MB/64MB aprox 4 blocks?
> > 3. How many task mappers will created? It is the same number as the
> number of blocks?
> > 4. If 4 mappers will be created, then one mapper will process the single
> value of the file, and the other three are just created and stopped?
> >
> > I look forward for your answers.
> > Thank you.
> > Regards,
> >  Florin
> >
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message