hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Quick question
Date Fri, 18 Feb 2011 19:25:27 GMT
The input is effectively split by lines, but under the covers, the actual
splits are by byte.  Each mapper will cleverly scan from the specified start
to the next line after the start point.  At then end, it will over-read to
the end of line that is at or after the end of its specified region.  This
can make the last split be a bit smaller than the others and the first be a
bit larger.

Practically speaking, however, your 2000 line file is extremely unlikely to
be split at all because it is sooo small.

On Fri, Feb 18, 2011 at 11:14 AM, maha <maha@umail.ucsb.edu> wrote:

> Hi all,
>  I want to check if the following statement is right:
>  If I use TextInputFormat to process a text file with 2000 lines (each
> ending with \n) with 20 mappers. Then each map will have a sequence of
> In other words,  the input is not split byte-wise but by lines.
> Is that right?
> Thank you,
> Maha

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message