hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
Date Thu, 23 May 2013 02:43:22 GMT

    [ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664808#comment-13664808

Bikas Saha commented on YARN-664:

I don't think that this is invalid as far as the API is concerned. There might be reasons
in the current implementations of the schedulers that may make supporting this prohibitive,
and so it was not done.
Say an application is running 2 concurrent but unrelated items that should not be blocked
by each other due to performance. Both have different resources sizes. If they are separated
by priority then the lower priority item will not get any containers until all the containers
for the higher priority item have been allocated. This is by definition of priority. That
may lead to poor performance. So the app would want to run both at the same priority and expect
the RM to allocate containers to both wherever resources are free to match them.
The most simple solution for the RM that I can think off the top of my head is to always use
the largest resource size. It may lead to fragmentation but is safe. Currently, the last resource
size is used. This means that if the client asks for 1GB and 2GB and the 1GB happens to be
the last one set by the RM then the RM will return 2 1GB containers, which is wrong.
If we choose to use the largest value as a solution, then we should close this as invalid
and use YARN-314 to implement that solution.
> throw InvalidRequestException for requests with different capabilities at the same priority
> -------------------------------------------------------------------------------------------
>                 Key: YARN-664
>                 URL: https://issues.apache.org/jira/browse/YARN-664
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
> Nothing stops an application from submitting a request with priority=1, location=*, memory=1024
and a request with priority=1, location=rack1, memory=2048.  However, this does not make sense
under the request model and can cause bad things to happen in the scheduler.  It should be
possible to detect this at AMRM heartbeat time and throw an exception.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message