tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-221) Deadlock in DAGSchedulerMRR when setting slow start to 0.5f on a small cluster
Date Thu, 15 Aug 2013 23:09:48 GMT

    [ https://issues.apache.org/jira/browse/TEZ-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741635#comment-13741635

Bikas Saha commented on TEZ-221:

While a bunch of these changes helped resolve deadlocks I dont see much value added by this
over the natural order scheduler. Both support slow start as maps complete and provide space
for reduces. DAGMRRScheduler allows a stronger reduce resource guarantees that allow reducers
to run even before some mappers even get to run. While this may be useful for some hand tuned
scenarios, it requires a tricky deadlock-prone dance with priorities and IMO should be left
as an expert level option. Also, making MRR logic work in a DAG gets even trickier. I plan
to keep this scheduler in the code but default to using natural order scheduler. If needed
this MRR scheduler may be plugged in via to-be-added API/config.
> Deadlock in DAGSchedulerMRR when setting slow start to 0.5f on a small cluster
> ------------------------------------------------------------------------------
>                 Key: TEZ-221
>                 URL: https://issues.apache.org/jira/browse/TEZ-221
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: yeshavora
>            Assignee: Bikas Saha
>              Labels: TEZ-0.2.0
>         Attachments: TEZ-221.1.patch, TEZ-221.2.patch, TEZ-221.3.patch, TEZ-221.4.preemption.patch,
> generate 10GB data using randomwriter and run sort job using tez-mapreduce-example.jar
on single node cluster. The job runs very slow. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message