tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-3984) Shuffle: Out of Band DME event sending causes errors
Date Mon, 25 Jan 2021 23:25:00 GMT

    [ https://issues.apache.org/jira/browse/TEZ-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271724#comment-17271724
] 

Hadoop QA commented on TEZ-3984:
--------------------------------

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 49s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  0s{color} |
{color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} branch-0.9 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m  8s{color}
| {color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 28s{color} |
{color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 16s{color}
| {color:green} branch-0.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 33s{color} |
{color:green} branch-0.9 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 35s{color} | {color:blue}
Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 32s{color} |
{color:green} branch-0.9 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 17s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 18s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 18s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 18s{color}
| {color:orange} tez-runtime-library: The patch generated 1 new + 241 unchanged - 0 fixed
= 242 total (was 241) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 19s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  0s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 29s{color} | {color:green}
tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 16s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 30s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/artifact/out/Dockerfile
|
| JIRA Issue | TEZ-3984 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12939327/TEZ-3984.2-branch-0.9.patch
|
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
|
| uname | Linux 200d7de5c6c5 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | branch-0.9 / d6978fe16 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/artifact/out/diff-checkstyle-tez-runtime-library.txt
|
|  Test Results | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/testReport/ |
| Max. process+thread count | 104 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/91/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |


This message was automatically generated.



> Shuffle: Out of Band DME event sending causes errors
> ----------------------------------------------------
>
>                 Key: TEZ-3984
>                 URL: https://issues.apache.org/jira/browse/TEZ-3984
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.8.4, 0.9.1, 0.10.0
>            Reporter: Gopal Vijayaraghavan
>            Assignee: Jaume M
>            Priority: Critical
>              Labels: correctness
>             Fix For: 0.10.1
>
>         Attachments: TEZ-3984-branch-0.9.patch, TEZ-3984.1.patch, TEZ-3984.2-branch-0.9.patch,
TEZ-3984.2.patch, TEZ-3984.3.patch, TEZ-3984.4.patch, TEZ-3984.5.patch, TEZ-3984.5.patch
>
>
> In case of a task Input throwing an exception, the outputs are also closed in the LogicalIOProcessorRuntimeTask.cleanup().
> Cleanup ignore all the events returned by output close, however if any output tries to
send an event out of band by directly calling outputContext.sendEvents(events), then those
events can reach the AM before the task failure is reported.
> This can cause correctness issues with shuffle since zero sized events can be sent out
due to an input failure and downstream tasks may never reattempt a fetch from the valid attempt.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message