Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANx3uAgACGowhnB7Jc+HS8Z3E7+S3J3g2+QxZ1L4sjjR05Y4dw@mail.gmail.com>
References: 
 <CANx3uAgACGowhnB7Jc+HS8Z3E7+S3J3g2+QxZ1L4sjjR05Y4dw@mail.gmail.com>
Date: Tue, 24 Jul 2012 10:46:28 -0400
Message-ID: 
 <CAENxBwxppEreXX_nKuVcYnnsJa=AN_JNvzo9imKSumCTQ+1O-g@mail.gmail.com>
Subject: Re: hadoop FileSystem.close()
From: Edward Capriolo <edlinuxguru@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

In all my experience you let FileSystem instances close themselves.

On Tue, Jul 24, 2012 at 10:34 AM, Koert Kuipers <koert@tresata.com> wrote:
> Since FileSystem is a Closeable i would expect code using it to be like
> this:
>
> FileSystem fs = path.getFileSystem(conf);
> try {
>     // do something with fs, such as read from the path
> } finally {
>     fs.close()
> }
>
> However i have repeatedly gotten into trouble with this approach. In one
> situation it turned out that when i closed a FileSystem other operations
> that were using their own FileSystems (pointing to the same real-world HDFS
> filesystem) also saw their FileSystems closed, leading to very confusing
> read and write errors. This led me to believe that FileSystem should never
> be closed since it seemed to act like some sort of Singleton. However now
> was just looking at some code (Hoop server, to be precise) and noticed that
> FileSystems were indeed closed, but they were always threadlocal. Is this
> the right approach?
>
> And if FileSystem is threadlocal, is this safe (assuming fs1 and fs2 could
> point to the same real-world filesystem)?
>
> FileSystem fs1 = path.getFileSystem(conf);
> try {
>     FileSystem fs2 = path.getFileSystem(conf);
>     try {
>     // do something with fs2, such as read from the path
>     } finally {
>       fs2.close()
>     }
>     // do something with fs1, such as read from the path (note, fs2 is
> closed here, and i wouldn't be surprised if fs1 by now is also closed given
> my experience)
> } finally {
>   fs1.close()
> }