hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Farrellee <m...@redhat.com>
Subject Re: Is FileSystem thread-safe?
Date Tue, 02 Apr 2013 11:41:24 GMT
If you're interested in the semantics of FileSystem operations, have a 
look a HADOOP-9371[0]

Depending on what you're trying to do, the thread-safety of a particular 
FS implementation in a single JVM instance may not be as important as 
the semantics you get across JVM instances.

Best,


matt

[0] https://issues.apache.org/jira/browse/HADOOP-9371


On 04/01/2013 06:17 PM, Arpit Agarwal wrote:
> Hi John,
>
> DistributedFileSystem is intended to be thread-safe, true to its name.
>
> Metadata operations are handled by the NameNode server which
> synchronizes concurrent client requests via locks (you can look at the
> FSNameSystem class).
>
> Some discussion on the thread-safety aspects of HDFS:
> http://storageconference.org/2010/Papers/MSST/Shvachko.pdf
>
> -Arpit
>
>
> On Sun, Mar 31, 2013 at 11:52 AM, Ted Yu <yuzhihong@gmail.com
> <mailto:yuzhihong@gmail.com>> wrote:
>
>     If you look at DistributedFileSystem source code, you would see that
>     it calls the DFSClient field member for most of the actions.
>     Requests to Namenode are then made through ClientProtocol.
>
>     An hdfs committer would be able to give you affirmative answer.
>
>
>     On Sun, Mar 31, 2013 at 11:27 AM, John Lilley
>     <john.lilley@redpoint.net <mailto:john.lilley@redpoint.net>> wrote:
>
>         *From:*Ted Yu [mailto:yuzhihong@gmail.com
>         <mailto:yuzhihong@gmail.com>]
>         *Subject:* Re: Is FileSystem thread-safe?____
>
>         >>FileSystem is an abstract class, what concrete class are you
>         using (DistributedFileSystem, etc) ? ____
>
>         Good point.  I am calling FileSystem.get(URI uri, Configuration
>         conf) with an URI like “hdfs://server:port/…” on a remote
>         server, so I assume it is creating a DistributedFileSystem.
>         However I am not finding any documentation discussing its
>         thread-safety (or lack thereof), perhaps you can point me to it?____
>
>         Thanks, john____
>
>
>


Mime
View raw message