hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
Date Wed, 25 Jul 2012 18:28:36 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422480#comment-13422480

Robert Joseph Evans commented on MAPREDUCE-4327:

I have looked through the patch somewhat quickly. 

My comments are mostly the same ones that I had for the previous patches.

I don't really like having the resource comparator class configuration be specific to the
scheduler. I would prefer to see it be available for both the fifo and the capacity scheduler.
This is very minor, but it enforces consistency between the schedulers.

The more I think about it the more I want to see the ability to request only part of a core.
 I don't think we need to make it a true float.  Perhaps we should round up to the closest
quarter of a core, but requiring full core increments is too course of a measure.  I think
we are going to get bad cluster utilization unless we can do a more fined grained approach.

Also inside LeafQueue.java and ParentQueue.java there is some code that was refactored to
use the new Resources Methods, but the original code is still there, just commented out. 
Please clean this up.

> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-4327
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, resourcemanager, scheduler
>    Affects Versions: 2.0.0-alpha
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>         Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch,
MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message