Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 21 Apr 2016 21:59:12 +0000 (UTC)
From: "Ben Podgursky (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12961400.1461275924000.1855.1461275952942@Atlassian.JIRA>
In-Reply-To: <JIRA.12961400.1461275924000@Atlassian.JIRA>
References: <JIRA.12961400.1461275924000@Atlassian.JIRA>
 <JIRA.12961400.1461275924582@arcas>
Subject: [jira] [Created] (HDFS-10323) transient deleteOnExit failure in
 ViewFileSystem due to close() ordering
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ben Podgursky created HDFS-10323:
------------------------------------

             Summary: transient deleteOnExit failure in ViewFileSystem due =
to close() ordering
                 Key: HDFS-10323
                 URL: https://issues.apache.org/jira/browse/HDFS-10323
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: federation
            Reporter: Ben Podgursky


After switching to using a ViewFileSystem, fs.deleteOnExit calls began fail=
ing frequently, displaying this error on failure:

16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for =
path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84

Since FileSystem eats the error involved, it is difficult to be sure what t=
he error is, but I believe what is happening is that the ViewFileSystem=E2=
=80=99s child FileSystems are being close()=E2=80=99d before the ViewFileSy=
stem, due to the random order ClientFinalizer closes FileSystems; so then w=
hen the ViewFileSystem tries to close(), it tries to forward the delete() c=
alls to the appropriate child, and fails because the child is already close=
d.

I=E2=80=99m unsure how to write an actual Hadoop test to reproduce this, si=
nce it involves testing behavior on actual JVM shutdown.  However, I can ve=
rify that while

{code:java}
fs.deleteOnExit(randomTemporaryDir);=E2=80=A8
{code}

regularly (~50% of the time) fails to delete the temporary directory, this =
code:

{code:java}
ViewFileSystem viewfs =3D (ViewFileSystem)fs1;=E2=80=A8for (FileSystem file=
System : viewfs.getChildFileSystems()) {=E2=80=A8  if (fileSystem.exists(ra=
ndomTemporaryDir)) {=E2=80=A8    fileSystem.deleteOnExit(randomTemporaryDir=
);=E2=80=A8  }=E2=80=A8}=E2=80=A8
{code}

always successfully deletes the temporary directory on JVM shutdown.

I am not very familiar with FileSystem inheritance hierarchies, but at firs=
t glance I see two ways to fix this behavior:

1)  ViewFileSystem could forward deleteOnExit calls to the appropriate chil=
d FileSystem, and not hold onto that path itself.

2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then al=
l other FileSystems. =20

Would appreciate any thoughts of whether this seems accurate, and thoughts =
(or help) on the fix.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)