tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-344) Add an option to allow node-local only scheduling in the TaskScheduler
Date Tue, 06 Aug 2013 06:26:49 GMT

    [ https://issues.apache.org/jira/browse/TEZ-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730427#comment-13730427
] 

Siddharth Seth commented on TEZ-344:
------------------------------------

bq. Whats the use case? Can the testing be done with a private build and avoid code change
"for now"? We all know there is nothing like "for now" 
Worst case, it's an indefinite "for now" :), but realistically this is something we should
fix sooner rather than later - i.e. delaying allocations in the AM scheduler to increase locality.
Changes like these tend to get lost in private builds.  Being able to figure out the impact
of locality on different environments is a useful performance tool.
YARN would've been an option, but I don't want to be debugging YARN scheduler allocations
while working on Tez. YARN white-listing isn't issue free - and should likely come with it's
own warning about usage.
                
> Add an option to allow node-local only scheduling in the TaskScheduler
> ----------------------------------------------------------------------
>
>                 Key: TEZ-344
>                 URL: https://issues.apache.org/jira/browse/TEZ-344
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>              Labels: TEZ-0.2.0
>
> This, for now, is primarily to help with testing of Tez on clusters.
> Would have to go in with a warning since this could cause jobs to hang / run for a long
time.
> Longer term, this can be enhanced to set limits on how long to wait before assigning
non-local tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message