tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Hussein (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (TEZ-4067) Tez Speculation decision is calculated on each update by the dispatcher
Date Thu, 07 Nov 2019 21:08:00 GMT

    [ https://issues.apache.org/jira/browse/TEZ-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969556#comment-16969556
] 

Ahmed Hussein edited comment on TEZ-4067 at 11/7/19 9:07 PM:
-------------------------------------------------------------

Uploaded a new patch to fix error reported in checkstyle and findbugs.


was (Author: ahussein):
Uploaded a new patch to fix error reported in checkstyle and windbags.

> Tez Speculation decision is calculated on each update by the dispatcher
> -----------------------------------------------------------------------
>
>                 Key: TEZ-4067
>                 URL: https://issues.apache.org/jira/browse/TEZ-4067
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Ahmed Hussein
>            Assignee: Ahmed Hussein
>            Priority: Minor
>         Attachments: TEZ-4067.001.patch, TEZ-4067.002.patch, TEZ-4067.003.patch
>
>
> LegacySpeculator is an object field in VertexImpl. Therefore, all events are handled
synchronously by the caller (dispatcher). This implies the following:
>  # the dispatcher spends long time executing updateStatus as it needs to check the runtime
estimation of the tezAttempts within the vertex.
>  # the speculator is per stage: lunching a speculation may not the optimum decision.
Ideally, based on resources, speculated tasks should be the ones with slowest progress.
>  # the time between speculation is skewed because there is a big delay for the dispatcher
to complete a full cycle. Also, speculation will be more aggressive compared to MR because
MR waits for "soonest.retry.after.speculate" whenever a task is speculated. On the other hand,
Tez speculates more tasks as it processes stages in parallel.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message