hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Trezzo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6690) Limit the number of resources a single map reduce job can submit for localization
Date Fri, 06 May 2016 20:15:12 GMT
Chris Trezzo created MAPREDUCE-6690:

             Summary: Limit the number of resources a single map reduce job can submit for
                 Key: MAPREDUCE-6690
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
            Reporter: Chris Trezzo
            Assignee: Chris Trezzo

Users will sometimes submit a large amount of resources to be localized as part of a single
map reduce job. This can cause issues with YARN localization that destabilize the cluster
and potentially impact other user jobs. These resources are specified via the files, libjars,
archives and jobjar command line arguments or directly through the configuration (i.e. distributed
cache api). The resources specified could be too large in multiple dimensions:
# Total size
# Number of files
# Size of an individual resource (i.e. a large fat jar)

We would like to encourage good behavior on the client side by having the option of enforcing
resource limits along the above dimensions.

There should be a separate effort to enforce limits at the YARN layer on the server side,
but this jira is only covering the map reduce layer on the client side. In practice, having
these client side limits will get us a long way towards preventing these localization anti-patterns.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message