hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9600) do not check replication if the block is under construction
Date Wed, 30 Dec 2015 10:06:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074893#comment-15074893
] 

Hadoop QA commented on HDFS-9600:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 32s {color}
| {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 42s {color} | {color:red}
hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 44s {color} | {color:red}
hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color}
| {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s {color} |
{color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 20s {color}
| {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 54s {color} | {color:red}
hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 272 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s {color} |
{color:green} branch-2.6 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s {color} |
{color:green} branch-2.6 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 39s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 39s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 43s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 43s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s {color} | {color:red}
The patch has 2664 line(s) that end in whitespace. Use git apply --whitespace=fix. {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 10s {color} | {color:red}
The patch has 95 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 52s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green}
the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 41s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 43s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 36s {color} | {color:red}
Patch generated 73 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 57s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:date2015-12-30 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12779934/HDFS-9600-branch-2.6.patch
|
| JIRA Issue | HDFS-9600 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux f985a816dd21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.6 / 0f9dd48 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
|
| compile | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| compile | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
|
| findbugs | v1.3.9 |
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| compile | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| javac | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| compile | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
|
| javac | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-eol.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-tabs.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
|
| JDK v1.7.0_91  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Max memory used | 71MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/14011/console |


This message was automatically generated.



> do not check replication if the block is under construction
> -----------------------------------------------------------
>
>                 Key: HDFS-9600
>                 URL: https://issues.apache.org/jira/browse/HDFS-9600
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Phil Yang
>            Assignee: Phil Yang
>            Priority: Critical
>         Attachments: HDFS-9600-branch-2.6.patch, HDFS-9600-branch-2.7.patch, HDFS-9600-branch-2.patch,
HDFS-9600-v1.patch, HDFS-9600-v2.patch, HDFS-9600-v3.patch, HDFS-9600-v4.patch
>
>
> When appending a file, we will update pipeline to bump a new GS and the old GS will be
considered as out of date. When changing GS, in BlockInfo.setGenerationStampAndVerifyReplicas
we will remove replicas having old GS which means we will remove all replicas because no DN
has new GS until the block with new GS is added to blockMaps again by DatanodeProtocol.blockReceivedAndDeleted.
> If we check replication of this block before it is added back, it will be regarded as
missing. The probability is low but if there are decommissioning nodes the DecommissionManager.Monitor
will scan all blocks belongs to decommissioning nodes with a very fast speed so the probability
of finding missing block is very high but actually they are not missing. 
> Furthermore, after closing the appended file, in FSNamesystem.finalizeINodeFileUnderConstruction,
it will checkReplication. If some of nodes are decommissioning, this block with new GS will
be added to UnderReplicatedBlocks map so there are two blocks with same ID in this map, one
is in QUEUE_WITH_CORRUPT_BLOCKS and the other is in QUEUE_HIGHEST_PRIORITY or QUEUE_UNDER_REPLICATED.
And there will be many missing blocks warning in NameNode website but there is no corrupt
files...
> Therefore, I think the solution is we should not check replication if the block is under
construction. We only check complete blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message