hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: mapreduce.job.max.split.locations just a warning in hadoop 1.0.3 but not in 2.0.1-alpha?
Date Thu, 14 Jun 2012 15:02:43 GMT
Hey Jim,

These are limits on the locations of a single split (locations for a
regular File would mean where all the file split's blocks may reside).
They do not control or cap inputs, just cap the maximum number of
locations shippable per InputSplit object. For a 'regular' job on a
'regular' cluster, the input split's locations would only be 3 at most
(3 replica locations to data-localize to). Its very rare to hit this
limit and this limit exists only to prevent abuse of memory (One may
send across lots of string values via the locations array via their
input format, say). It also helps catch stupid mistakes on custom
location generation, as an added bonus I guess.

I personally do know not why MAPREDUCE-4146 decided to throw
IOException but I think thats a better idea for a user-set limit than
to just log a warn and cap it to a size. Please raise it on the JIRA
to get the author's/reporters' notice! May have just been a foresight,
but I think I like it this way (gives a proper 'limit' enforcement
feel) :)

On Thu, Jun 14, 2012 at 6:11 PM, Jim Donofrio <donofrio111@gmail.com> wrote:
> I didnt hear anything from common-user about this, maybe that was the wrong
> list because this is more a development issue.
>
>
>        final int max_loc = conf.getInt(MAX_SPLIT_LOCATIONS, 10);
>        if (locations.length > max_loc) {
>          LOG.warn("Max block location exceeded for split: "
>              + split + " splitsize: " + locations.length +
>              " maxsize: " + max_loc);
>          locations = Arrays.copyOf(locations, max_loc);
>        }
>
> I was wondering about the above code in JobSplitWriter in hadoop 1.0.3.
> The below commit comment is somewhat vague. I saw MAPREDUCE-1943 about
> setting limits to save memory on the jobtracker. I wanted to confirm
> that the above fix just serves as a warning and saves memory on the
> jobtracker and does not cap the input at all since most inputformats
> seem to ignore the locations?
>
> I also wanted to know why the recent MAPREDUCE-4146 added this cap to
> 2.0.1-alpha but with the original capping behavior of causing the job to
> fail by throwing an IOException instead of just warning the user as the
> current code does?
>
>
> commit 51be5c3d61cbc7960174493428fbaa41d5fbe84d
> Author: Chris Douglas <cdouglas@apache.org>
> Date:   Fri Oct 1 01:49:51 2010 -0700
>
>     Change client-side enforcement of limit on locations per split
>    to be advisory. Truncate on client, optionally fail job at
> JobTracker if
>    exceeded. Added mapreduce.job.max.split.locations property.
>
>    +++ b/YAHOO-CHANGES.txt
>    +     Change client-side enforcement of limit on locations per split
>    +    to be advisory. Truncate on client, optionally fail job at
> JobTracker if
>    +    exceeded. Added mapreduce.job.max.split.locations property.
> (cdouglas)
>    +
>



-- 
Harsh J

Mime
View raw message