Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CANx3uAiK5BuPFZpPKchJBz68N3yyWsMQe89bD+bfTpS7Qy=qUA@mail.gmail.com>
References: 
 <CANx3uAiK5BuPFZpPKchJBz68N3yyWsMQe89bD+bfTpS7Qy=qUA@mail.gmail.com>
Date: Mon, 6 Aug 2012 13:32:36 -0400
Message-ID: 
 <CANx3uAhWyD3mi4wyNFyyUmMTibA2jdJJ6aS8BF8MGaEMYbFcBQ@mail.gmail.com>
Subject: Fwd: fs cache giving me headaches
From: Koert Kuipers <koert@tresata.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d043c06b67b836404c69c433c

--f46d043c06b67b836404c69c433c
Content-Type: text/plain; charset=ISO-8859-1

---------- Forwarded message ----------
From: "Koert Kuipers" <koert@tresata.com>
Date: Aug 4, 2012 1:54 PM
Subject: fs cache giving me headaches
To: <common-user@hadoop.apache.org>

nothing has confused me as much in hadoop as FileSystem.close().
any decent java programmer that sees that an object implements Closable
writes code like this:
Final FileSystem fs = FileSystem.get(conf);
try {
    // do something with fs
} finally {
    fs.close();
}

so i started out using hadoop FileSystem like this, and i ran into all
sorts of weird errors where FileSystems in unrelated code (sometimes not
even my code) started misbehaving and streams where unexpectedly shut. Then
i realized that FileSystem uses a cache and close() closes it for everyone!
Not pretty in my opinion, but i can live with it. So i checked other code
and found that basically nobody closes FileSystems. Apparently the expected
way of using FileSystems is to simple never close them. So i adopted this
approach (which i think is really contrary to java conventions for a
Closeable).

Lately i started working on some code for a daemon/server where many
FileSystems objects are created for different users (UGIs) that use the
service. As it turns out other projects have run into trouble with the
FileSystem cache in situations like this (for example, Scribe and Hoop). I
imagine the cache can get very large and cause problems (i have not tested
this myself).

Looking at the code for Hoop i noticed they simply turned off the
FileSystem cache and made sure to close every FileSystem. So here the
suggested approach to deal with FileSystems seems to be:
Final FileSystem fs = FileSystem.newInstance(conf); // or
FileSystem.get(conf) but with caching turned off in the conf
try {
    // do something with fs
} finally {
    fs.close();
}

This code bypasses the cache if i understand it correctly, avoiding any
cache size limitations. However if i adopt this approach i basically can
not re-use any existing code or libraries that do not close FileSystems,
splitting the codebase into two which is pretty ugly. And this code is not
efficient in situations where there are very few used FileSystem objects
and a cache would improve performance, so the split works both ways.

In short, there is so single way to code with FileSystem that works in both
situations! Ideally i would have liked fs.close() to do the right thing
depending in the settings: if cache is turned off it closes the FileSystem,
and if it is turned on its a NOOP. That way i could always use
FileSystem.get(conf) and always close my filesystems, and the code would be
usable irrespective of whether the cache is turned on or off.

Any insights or suggestions? Thanks!

--f46d043c06b67b836404c69c433c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote">---------- Forwarded message ----------<br>From:=
 &quot;Koert Kuipers&quot; &lt;<a href=3D"mailto:koert@tresata.com">koert@t=
resata.com</a>&gt;<br>Date: Aug 4, 2012 1:54 PM<br>Subject: fs cache giving=
 me headaches<br>
To:  &lt;<a href=3D"mailto:common-user@hadoop.apache.org">common-user@hadoo=
p.apache.org</a>&gt;<br><br type=3D"attribution">nothing has confused me as=
 much in hadoop as FileSystem.close().<br>any decent java programmer that s=
ees that an object implements Closable writes code like this:<br>
Final FileSystem fs =3D FileSystem.get(conf);<br>try {<br>
=A0=A0=A0 // do something with fs<br>} finally {<br>=A0=A0=A0 fs.close();<b=
r>}<br><br>so i started out using hadoop FileSystem like this, and i ran in=
to all sorts of weird errors where FileSystems in unrelated code (sometimes=
 not even my code) started misbehaving and streams where unexpectedly shut.=
 Then i realized that FileSystem uses a cache and close() closes it for eve=
ryone! Not pretty in my opinion, but i can live with it. So i checked other=
 code and found that basically nobody closes FileSystems. Apparently the ex=
pected way of using FileSystems is to simple never close them. So i adopted=
 this approach (which i think is really contrary to java conventions for a =
Closeable).<br>

<br>Lately i started working on some code for a daemon/server where many Fi=
leSystems objects are created for different users (UGIs) that use the servi=
ce. As it turns out other projects have run into trouble with the FileSyste=
m cache in situations like this (for example, Scribe and Hoop). I imagine t=
he cache can get very large and cause problems (i have not tested this myse=
lf).<br>

<br>Looking at the code for Hoop i noticed they simply turned off the FileS=
ystem cache and made sure to close every FileSystem. So here the suggested =
approach to deal with FileSystems seems to be:<br>Final FileSystem fs =3D F=
ileSystem.newInstance(conf); // or FileSystem.get(conf) but with caching tu=
rned off in the conf<br>

try {<br>
=A0=A0=A0 // do something with fs<br>
} finally {<br>
=A0=A0=A0 fs.close();<br>
}<br>
<br>This code bypasses the cache if i understand it correctly, avoiding any=
 cache size limitations. However if i adopt this approach i basically can n=
ot re-use any existing code or libraries that do not close FileSystems, spl=
itting the codebase into two which is pretty ugly. And this code is not eff=
icient in situations where there are very few used FileSystem objects and a=
 cache would improve performance, so the split works both ways.<br>

<br>In short, there is so single way to code with FileSystem that works in =
both situations! Ideally i would have liked fs.close() to do the right thin=
g depending in the settings: if cache is turned off it closes the FileSyste=
m, and if it is turned on its a NOOP. That way i could always use FileSyste=
m.get(conf) and always close my filesystems, and the code would be usable i=
rrespective of whether the cache is turned on or off.<br>

<br>Any insights or suggestions? Thanks!<br><br>
</div>

--f46d043c06b67b836404c69c433c--