[ https://issues.apache.org/jira/browse/TEZ-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731293#comment-13731293
]
Siddharth Seth commented on TEZ-344:
------------------------------------
Haven't followed strict locality in detail on YARN.
For the following ask
T1 - host1, host2 | 1 container | strict locality
T2 - host3, host4 | 1 container | strict locality
Both the requests go as part of a single heartbeat.
Will the RM always guarantee 1 container for T1, and 1 for T2 - or is it possible for the
RM to allocate 1 container on host1 and 1 on host2 (effectively ignoring the T2 ask - since
I'm guessing there's no separation in the ask). From what I understand, and this could be
wrong, the RM could end up allocating on host1 and host2.
The config - for now - is forcing node locality (by returning containers which cannot be allocated
with node-locality). Like I mentioned earlier, this can evolve in the future to be a delay
before assigning a node local container.
bq. If its going to enforce node locality (instead of using YARN API's) then I dont think
its an approach I would like to take in the TaskScheduler.
More details on why this is unreasonable to commit would be useful. If YARN does allocation
for T1,T2 properly - then yes, this could be avoided in favor of a delay config at a later
point.
> Add an option to allow node-local only scheduling in the TaskScheduler
> ----------------------------------------------------------------------
>
> Key: TEZ-344
> URL: https://issues.apache.org/jira/browse/TEZ-344
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Labels: TEZ-0.2.0
>
> This, for now, is primarily to help with testing of Tez on clusters.
> Would have to go in with a warning since this could cause jobs to hang / run for a long
time.
> Longer term, this can be enhanced to set limits on how long to wait before assigning
non-local tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
|