hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
Date Tue, 09 Feb 2016 06:25:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138416#comment-15138416
] 

Hadoop QA commented on HADOOP-12780:
------------------------------------

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 5s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 16s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s {color} | {color:green}
hadoop-azure in the patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 22s {color} | {color:green}
hadoop-azure in the patch passed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 45s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12786995/HADOOP-12780.001.patch
|
| JIRA Issue | HADOOP-12780 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 065bbaccf7e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / acac729 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
|
| findbugs | v3.0.0 |
| JDK v1.7.0_91  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/testReport/
|
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Max memory used | 76MB |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> During atomic rename handle crash when one directory has been renamed but not file under
it.
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-12780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12780
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure
>    Affects Versions: 2.8.0
>            Reporter: madhumita chakraborty
>            Assignee: madhumita chakraborty
>            Priority: Critical
>         Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change to a metadata
file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file innerFile
> Content of the –renamePending.json file will be
> {   OldFolderName: parent/ folderToRename",
>      NewFolderName: "parent/renamedFolder", 
>      FileList: [ "innerFolder", "innerFolder/innerFile" ]
>  }
> Atfirst we rename all files within the source directory and then rename the source directory
at the last step
> The steps are
> 1.	Atfirst we will rename innerFolder, 
> 2.	Then rename innerFolder/innerFile 
> 3.	Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed. 
> Note that Azure storage does not natively support folder. So if a directory created by
mkdir command, we create an empty placeholder blob with metadata for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder has been renamed.
> When the process comes up, in redo path it will go through the –renamePending.json
file try to redo the renames. 
> For each file in file list of renamePending file it checks if the source file exists,
if source file exists then it renames the file. When it gets innerFolder, it calls filesystem.exists(innerFolder).
Now filesystem.exists(innerFolder) will return true, because file under that folder exists
even though the empty blob corresponding th that folder does not exist. So it will try to
rename this folder, and as the empty blob has already been deleted so this fails with exception
that “source blob does not exist”.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message