Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <49EBDCA3.2010006@comcast.net>
References: <49EBDCA3.2010006@comcast.net>
From: Aaron Kimball <aaron@cloudera.com>
Date: Sun, 19 Apr 2009 22:18:21 -0700
Message-ID: <d6d7c4410904192218q2e9960deh510d8a8015539727@mail.gmail.com>
Subject: Re: Are SequenceFiles split? If so, how?
To: core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369204308f65bf0467f5a9c2

--0016369204308f65bf0467f5a9c2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Yes, there can be more than one InputSplit per SequenceFile. The file will
be split more-or-less along 64 MB boundaries. (the actual "edges" of the
splits will be adjusted to hit the next block of key-value pairs, so it
might be a few kilobytes off.)

The SequenceFileInputFormat regards mapred.map.tasks (conf.setNumMapTasks())
as a hint, not a set-in-stone metric. (The number of reduce tasks, though,
is always 100% user-controlled.) If you need exact control over the number
of map tasks, you'll need to subclass it and modify this behavior. That
having been said -- are you sure you actually need to precisely control this
value? Or is it enough to know how many splits were created?

- Aaron

On Sun, Apr 19, 2009 at 7:23 PM, Barnet Wagman <b.wagman@comcast.net> wrote:

> Suppose a SequenceFile (containing keys and values that are BytesWritable)
> is used as input. Will it be divided into InputSplits?  If so, what's the
> criteria use for splitting?
>
> I'm interested in this because I need to control the number of map tasks
> used, which (if I understand it correctly), is equal to the number of
> InputSplits.
>
> thanks,
>
> bw
>

--0016369204308f65bf0467f5a9c2--