hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality
Date Fri, 22 Feb 2013 21:14:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584703#comment-13584703
] 

Alejandro Abdelnur commented on YARN-392:
-----------------------------------------

I'd like to restate he problem.

Making things a bit more high level, the end goal is for an AM to give certain hints to the
RM scheduler on how it plans to use requested resources.

Hints are just that, 'hints'. They may not be taken into consideration, AMs must not rely
on hints to be able to work properly. It is fine if the RM scheduler ignores hints completely
(because it is too busy or because it does not understand them). An RM scheduler that understands
a hint may use it to make more optimal allocation decisions and may give AMs a speed boost.

Another thing to keep in mind is that hints won't complicate the RM logic as the RM only involvement
is passing them to the scheduler.

Examples of hints are: gang scheduling, desired locality, desired multi-locality, resources
fulfillment timeout, future resource allocation.

I can understand the worries about going task centric, but I think the hints approach is a
bit different. Being able to specify hints will enable scheduling features experimentation
without requiring protocol changes. Eventually, if we find that a hint is a good feature to
support at scheduler API level we may eventually add it to the protocol/API.

The changes in the protocol/API would be as simple as having and extra String field in resource
requests and resources allocations to indicate hints (on requests) and receive the hints taken
into consideration (on allocations).

Thoughts?
                
> Make it possible to schedule to specific nodes without dropping locality
> ------------------------------------------------------------------------
>
>                 Key: YARN-392
>                 URL: https://issues.apache.org/jira/browse/YARN-392
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Sandy Ryza
>         Attachments: YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes and nowhere
else. The RM automatically relaxes locality to rack and * and assigns non-specified machines
to the app.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message