hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
Date Thu, 05 Jul 2012 17:24:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407294#comment-13407294

Robert Joseph Evans commented on MAPREDUCE-4327:

Andrew, Sorry this has taken me so long to get to this.  Thanks to both you and Arun for taking
this up.  It is something that is going to be really great when it is done.

I have a few comments on the code.

# ResourceComparator.java needs to have the Apache License Header in it.
# I don't really like having the resource comparator class configuration be specific to the
scheduler. I would prefer to see it be available for both the fifo and the capacity scheduler.
 This is very minor, but it enforces consistency between the schedulers.
# In a few places like SchedulerNode, we still operate on each of the resources separately.
 {code}     this.availableResource.setMemory(node.getTotalCapability().getMemory());
{code} It would really be nice to be able to abstract some of that away, like with the comparisons,
so that if we add in new resources in the future, we do not need to change the code again.
(This is also very minor)
# To answer your question about computeSlotMillis, I would say no, but I am open to others
opinions on this.  This counter is there to try an maintain backwards compatibility.  It was
intended to indicate how much total resources the job used, i.e. how many milliseconds this
job held a slot that no-one else could use.  Because there are no real slots any more, I would
prefer to see this metric deprecated, and replaced with something that breaks it down by the
resources involved.  But that is probably for a separate JIRA, because it is a potentially
complex question.
# I would like better protections against someone passing in a 0 or null for the number of
CPU cores in YARN.  For MR I see a new default for the number of CORES being requested, but
I don't see an equivalent in just YARN.  This is mostly because CPU cores is being added in
and I can see other applications, like the distributed shell, not being updated, which could
result in all kinds of issues.  It would be great that if no CPU request is given we default
to 1.
# Could we change the name of ResourceMemoryCpuComparator to something more like DefaultMultiResourceComparator?
 I think ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it is
the direction we are headed in. 
# Do we want to be able to schedule only part of a core (make the resource a float not an
int)?  For a Map or Reduce task we typically are only going to want 1 CPU, but some things
like the MR AM, unless it is a very big job, 0.5 cores is overkill for what it does.

This is just a cursory look but I like what I see.

To chime in on some of your questions
bq. Are you planning to change the definition of a queue's capacity?
I think this could be something very useful, but should probably be done on a separate JIRA.
bq. Do you plan to change how spare capacity is allocated?
What do you mean by spare capacity?  Do you mean capacity that is not currently in use?  If
so I would love to see a patch that does this, so that I can run gridmix on it both ways and
see what the results are.
bq. Are you planning to support priorities or weights within the queues?
I would also like to see something like this happen, but from discussions I have had in the
past, at least for the MRV1 case we would need something like preemption to be able to avoid
some potential deadlocks.  I could be wrong about that here, because the resource allocation
now behaves differently in with respect to a priority, but either way I think that discussion
is something that should be done on a separate JIRA to avoid blocking this coming in.

Thanks again to both you and Arun for doing this.
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-4327
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, resourcemanager, scheduler
>    Affects Versions: 2.0.0-alpha
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>         Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch,
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message