tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-344) Add an option to allow node-local only scheduling in the TaskScheduler
Date Tue, 06 Aug 2013 07:00:51 GMT

    [ https://issues.apache.org/jira/browse/TEZ-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730453#comment-13730453
] 

Bikas Saha commented on TEZ-344:
--------------------------------

Why should we do second level of delay scheduling in Tez when YARN already does that? In any
case, I dont think the intention is to add delay scheduling in this jira so lets not be distracted
by that. 

I am +1 to enable strict locality. We should be using YARN API's to achieve that. That way
we know that the right thing will happen automatically for maps and reduces. AMRMClient already
take care of the burden of inferring racks implicitly. Later, in getMatchingRequests() AMRMClient
understands which requests were made with strict locality and doesnt return those requests
when matches are asked for rack or *. So life is a lot simpler and correct for us.
Thinking more, since strict locality is something thats probably vertex specific, should it
be specified via DAG Vertex API?
                
> Add an option to allow node-local only scheduling in the TaskScheduler
> ----------------------------------------------------------------------
>
>                 Key: TEZ-344
>                 URL: https://issues.apache.org/jira/browse/TEZ-344
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>              Labels: TEZ-0.2.0
>
> This, for now, is primarily to help with testing of Tez on clusters.
> Would have to go in with a warning since this could cause jobs to hang / run for a long
time.
> Longer term, this can be enhanced to set limits on how long to wait before assigning
non-local tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message