tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Turner Eagles (Jira)" <j...@apache.org>
Subject [jira] [Updated] (TEZ-3984) Shuffle: Out of Band DME event sending causes errors
Date Mon, 25 Jan 2021 22:49:00 GMT

     [ https://issues.apache.org/jira/browse/TEZ-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Turner Eagles updated TEZ-3984:
----------------------------------------
    Fix Version/s:     (was: 0.10.0)
                   0.10.1

> Shuffle: Out of Band DME event sending causes errors
> ----------------------------------------------------
>
>                 Key: TEZ-3984
>                 URL: https://issues.apache.org/jira/browse/TEZ-3984
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.8.4, 0.9.1, 0.10.0
>            Reporter: Gopal Vijayaraghavan
>            Assignee: Jaume M
>            Priority: Critical
>              Labels: correctness
>             Fix For: 0.10.1
>
>         Attachments: TEZ-3984-branch-0.9.patch, TEZ-3984.1.patch, TEZ-3984.2-branch-0.9.patch,
TEZ-3984.2.patch, TEZ-3984.3.patch, TEZ-3984.4.patch, TEZ-3984.5.patch, TEZ-3984.5.patch
>
>
> In case of a task Input throwing an exception, the outputs are also closed in the LogicalIOProcessorRuntimeTask.cleanup().
> Cleanup ignore all the events returned by output close, however if any output tries to
send an event out of band by directly calling outputContext.sendEvents(events), then those
events can reach the AM before the task failure is reported.
> This can cause correctness issues with shuffle since zero sized events can be sent out
due to an input failure and downstream tasks may never reattempt a fetch from the valid attempt.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message