hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9832) Erasure Coding: Improve exception handling in ErasureCodingWorker#ReconstructAndTransferBlock
Date Fri, 19 Feb 2016 06:38:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153789#comment-15153789
] 

Hadoop QA commented on HDFS-9832:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 29s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} |
{color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s {color} |
{color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 54s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green}
trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s {color} |
{color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} |
{color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s {color} |
{color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s {color} | {color:green}
the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s {color} |
{color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 54m 20s {color} | {color:green}
hadoop-hdfs in the patch passed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 51m 53s {color} | {color:green}
hadoop-hdfs in the patch passed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 47s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12788606/HDFS-9832.001.patch
|
| JIRA Issue | HDFS-9832 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 8be53b7e3573 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 748b6c0 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
|
| findbugs | v3.0.0 |
| JDK v1.7.0_95  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/14544/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/14544/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Erasure Coding: Improve exception handling in ErasureCodingWorker#ReconstructAndTransferBlock
> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9832
>                 URL: https://issues.apache.org/jira/browse/HDFS-9832
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding
>            Reporter: Lin Yiqun
>            Assignee: Lin Yiqun
>            Priority: Minor
>             Fix For: 3.0.0
>
>         Attachments: HDFS-9832.001.patch
>
>
> There are two places in {{ErasureCodingWorker#ReconstructAndTransferBlock}} that I think
can be improved.
> 1.In run method, the step3 transfer data will be failed sometimes, and this will cause
buffers not be cleared completely, is better to invoke clearBuffer again in finally handling?
> {code}
>         while (positionInBlock < maxTargetLength) {
>           final int toReconstruct = (int) Math.min(
>               bufferSize, maxTargetLength - positionInBlock);
>           // step1: read from minimum source DNs required for reconstruction.
>           // The returned success list is the source DNs we do real read from
>           Map<ExtendedBlock, Set<DatanodeInfo>> corruptionMap = new HashMap<>();
>           try {
>             success = readMinimumStripedData4Reconstruction(success,
>                 toReconstruct, corruptionMap);
>           } finally {
>             // report corrupted blocks to NN
>             reportCorruptedBlocks(corruptionMap);
>           }
>           // step2: decode to reconstruct targets
>           reconstructTargets(success, targetsStatus, toReconstruct);
>           // step3: transfer data
>           if (transferData2Targets(targetsStatus) == 0) {
>             String error = "Transfer failed for all targets.";
>             throw new IOException(error);
>           }
>           clearBuffers();
>           positionInBlock += toReconstruct;
>         }
> {code}
> 2.Is better to set null to buffers objects, targetsOutput and socket objects in finally
handling code?
> {code}
>       } finally {
>         datanode.decrementXmitsInProgress();
>         // close block readers
>         for (StripedReader stripedReader : stripedReaders) {
>           closeBlockReader(stripedReader.blockReader);
>         }
>         for (int i = 0; i < targets.length; i++) {
>           IOUtils.closeStream(targetOutputStreams[i]);
>           IOUtils.closeStream(targetInputStreams[i]);
>           IOUtils.closeStream(targetSockets[i]);
>         }
>       }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message