hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes, fix allocation configurations and document them
Date Mon, 23 Apr 2012 15:37:35 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259668#comment-13259668
] 

Hudson commented on MAPREDUCE-3812:
-----------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2190 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2190/])
    MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document
them (Harsh J via bobby) (Revision 1329282)

     Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

                
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3812
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2, performance
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Harsh J
>         Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt,
MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and
MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM
slot size from 1 slot to two slots to work around the issues with AM heap. Now that those
are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense
as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such
values, we should lower the minimum alloc to 128 MB to allow special requests of low memory
out of the box itself. Shouldn't impact MR App in any way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message