Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Tue, 9 Feb 2016 19:50:18 +0000 (UTC)
From: "madhumita chakraborty (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12937478.1454933689000.8505.1455047418282@Atlassian.JIRA>
In-Reply-To: <JIRA.12937478.1454933689000@Atlassian.JIRA>
References: <JIRA.12937478.1454933689000@Atlassian.JIRA>
 <JIRA.12937478.1454933689288@arcas>
Subject: [jira] [Commented] (HADOOP-12780) During atomic rename handle crash
 when one directory has been renamed but not file under it.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HADOOP-12780?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15=
139632#comment-15139632 ]=20

madhumita chakraborty commented on HADOOP-12780:
------------------------------------------------

[~cnauroth] Could you please take a look at the patch?

> During atomic rename handle crash when one directory has been renamed but=
 not file under it.
> -------------------------------------------------------------------------=
-------------------
>
>                 Key: HADOOP-12780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12780
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure
>    Affects Versions: 2.8.0
>            Reporter: madhumita chakraborty
>            Assignee: madhumita chakraborty
>            Priority: Critical
>         Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed cha=
nge to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file=
 innerFile
> Content of the =E2=80=93renamePending.json file will be
> {   OldFolderName: parent/ folderToRename",
>      NewFolderName: "parent/renamedFolder",=20
>      FileList: [ "innerFolder", "innerFolder/innerFile" ]
>  }
> Atfirst we rename all files within the source directory and then rename t=
he source directory at the last step
> The steps are
> 1.=09Atfirst we will rename innerFolder,=20
> 2.=09Then rename innerFolder/innerFile=20
> 3.=09Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.=20
> Note that Azure storage does not natively support folder. So if a directo=
ry created by mkdir command, we create an empty placeholder blob with metad=
ata for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolde=
r has been renamed.
> When the process comes up, in redo path it will go through the =E2=80=93r=
enamePending.json file try to redo the renames.=20
> For each file in file list of renamePending file it checks if the source =
file exists, if source file exists then it renames the file. When it gets i=
nnerFolder, it calls filesystem.exists(innerFolder). Now filesystem.exists(=
innerFolder) will return true, because file under that folder exists even t=
hough the empty blob corresponding th that folder does not exist. So it wil=
l try to rename this folder, and as the empty blob has already been deleted=
 so this fails with exception that =E2=80=9Csource blob does not exist=E2=
=80=9D.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)