tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3984) Shuffle: Out of Band DME event sending causes errors
Date Mon, 27 Aug 2018 22:34:00 GMT
Gopal V created TEZ-3984:

             Summary: Shuffle: Out of Band DME event sending causes errors
                 Key: TEZ-3984
                 URL: https://issues.apache.org/jira/browse/TEZ-3984
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.9.1, 0.8.4, 0.10.0
            Reporter: Gopal V

In case of a task Input throwing an exception, the outputs are also closed in the LogicalIOProcessorRuntimeTask.cleanup().

Cleanup ignore all the events returned by output close, however if any output tries to send
an event out of band by directly calling outputContext.sendEvents(events), then those events
can reach the AM before the task failure is reported.

This can cause correctness issues with shuffle since zero sized events can be sent out due
to an input failure and downstream tasks may never reattempt a fetch from the valid attempt.

This message was sent by Atlassian JIRA

View raw message