tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-14) Support for speculation of slow tasks
Date Wed, 22 Oct 2014 18:27:33 GMT

    [ https://issues.apache.org/jira/browse/TEZ-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180286#comment-14180286
] 

Bikas Saha commented on TEZ-14:
-------------------------------

You are right. In the long term there may be cases where some customization may be needed.
This could either be done via configurable speculator or by involving the vertex manager since
the vertex manager does the original task scheduling. However, there are no concrete cases
now. Also, speculators would need stats etc from the system and we dont yet have a stats API
that exposes this data. Until then we might be better off not making that configurable and
using this basic speculator that works for the straightforward cases. That is why its currently
not configurable.
Also, once some of the stats work is done, I plan to replace this legacy speculator with a
more advanced version that understands source slowness, does pro-active cancellation etc like
described in the Mantri paper above. My guess is that a well-written default speculator will
remove the need for custom speculators.

Did you get a chance to review the code or try it out on a cluster?

> Support for speculation of slow tasks
> -------------------------------------
>
>                 Key: TEZ-14
>                 URL: https://issues.apache.org/jira/browse/TEZ-14
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: TEZ-14.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message