tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-344) Add an option to allow node-local only scheduling in the TaskScheduler
Date Tue, 06 Aug 2013 21:24:48 GMT

    [ https://issues.apache.org/jira/browse/TEZ-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731293#comment-13731293
] 

Siddharth Seth commented on TEZ-344:
------------------------------------

Haven't followed strict locality in detail on YARN.
For the following ask
T1 - host1, host2 | 1 container | strict locality
T2 - host3, host4 | 1 container | strict locality

Both the requests go as part of a single heartbeat.

Will the RM always guarantee 1 container for T1, and 1 for T2 - or is it possible for the
RM to allocate 1 container on host1 and 1 on host2 (effectively ignoring the T2 ask - since
I'm guessing there's no separation in the ask). From what I understand, and this could be
wrong, the RM could end up allocating on host1 and host2.

The config - for now - is forcing node locality (by returning containers which cannot be allocated
with node-locality). Like I mentioned earlier, this can evolve in the future to be a delay
before assigning a node local container.

bq. If its going to enforce node locality (instead of using YARN API's) then I dont think
its an approach I would like to take in the TaskScheduler.
More details on why this is unreasonable to commit would be useful. If YARN does allocation
for T1,T2 properly - then yes, this could be avoided in favor of a delay config at a later
point.
                
> Add an option to allow node-local only scheduling in the TaskScheduler
> ----------------------------------------------------------------------
>
>                 Key: TEZ-344
>                 URL: https://issues.apache.org/jira/browse/TEZ-344
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>              Labels: TEZ-0.2.0
>
> This, for now, is primarily to help with testing of Tez on clusters.
> Would have to go in with a warning since this could cause jobs to hang / run for a long
time.
> Longer term, this can be enhanced to set limits on how long to wait before assigning
non-local tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message