hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10763) Open files can leak permanently due to inconsistent lease update
Date Wed, 17 Aug 2016 22:52:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425542#comment-15425542
] 

Hadoop QA commented on HDFS-10763:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 21s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m  4s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  8s{color} |
{color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 11s{color} |
{color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 26s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  4s{color} |
{color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 15s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 58s{color} |
{color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 59s{color} |
{color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 40s{color} |
{color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 51s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 57s{color} |
{color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 57s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 59s{color} |
{color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 59s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 21s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 56s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 12s{color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  1s{color} | {color:red}
The patch has 1578 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>.
Refer https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m 42s{color} | {color:red}
The patch 78 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 13s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 58s{color} |
{color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 41s{color} |
{color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 40s{color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 24s{color} | {color:red}
The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m  3s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
|
|   | hadoop.hdfs.server.blockmanagement.TestBlockManager |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
| JDK v1.7.0_101 Failed junit tests | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
|
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.balancer.TestBalancer |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c420dfe |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12824195/HDFS-10763.branch-2.7.supplement.patch
|
| JIRA Issue | HDFS-10763 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux fccfd73e7905 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7 / 040a1b7 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
|
| findbugs | v3.0.0 |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/artifact/patchprocess/whitespace-eol.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/artifact/patchprocess/whitespace-tabs.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
|
| JDK v1.7.0_101  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16454/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Open files can leak permanently due to inconsistent lease update
> ----------------------------------------------------------------
>
>                 Key: HDFS-10763
>                 URL: https://issues.apache.org/jira/browse/HDFS-10763
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.3, 2.6.4
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 2.7.4, 3.0.0-alpha2
>
>         Attachments: HDFS-10763.br27.patch, HDFS-10763.branch-2.7.supplement.patch, HDFS-10763.patch
>
>
> This can heppen during {{commitBlockSynchronization()}} or a client gives up on closing
a file after retries.
> From {{finalizeINodeFileUnderConstruction()}}, the lease is removed first and then the
inode is turned into the closed state. But if any block is not in COMPLETE state, 
> {{INodeFile#assertAllBlocksComplete()}} will throw an exception. This will cause the
lease is removed from the lease manager, but not from the inode. Since the lease manager does
not have a lease for the file, no lease recovery will happen for this file. Moreover, this
broken state is persisted and reconstructed through saving and loading of fsimage. Since no
replication is scheduled for the blocks for the file, this can cause a data loss and also
block decommissioning of datanode.
> The lease cannot be manually recovered either. It fails with
> {noformat}
> ...AlreadyBeingCreatedException): Failed to RECOVER_LEASE /xyz/xyz for user1 on
>  0.0.0.1 because the file is under construction but no leases found.
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2950)
> ...
> {noformat}
> When a client retries {{close()}}, the same inconsistent state is created, but it can
work in the next time since {{checkLease()}} only looks at the inode, not the lease manager
in this case. The close behavior is different if HDFS-8999 is activated by setting {{dfs.namenode.file.close.num-committed-allowed}}
to 1 (unlikely) or 2 (never). 
> In principle, the under-construction feature of an inode and the lease in the lease manager
should never go out of sync. The fix involves two parts.
> 1) Prevent inconsistent lease updates. We can achieve this by calling {{removeLease()}}
after checking the block state. 
> 2) Avoid reconstructing inconsistent lease states from a fsimage. 1) alone does not correct
the existing inconsistencies surviving through fsimages.  This can be done during fsimage
loading time by making sure a corresponding lease exists for each inode that are with the
underconstruction feature. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message