hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: Is FileSystem thread-safe?
Date Fri, 17 May 2013 13:40:48 GMT
Thanks! Does this also imply that multiple clients may open the same HDFS file for append simultaneously,
and expect append requests to be interleaved?

From: Arpit Agarwal [mailto:aagarwal@hortonworks.com]
Sent: Monday, April 01, 2013 4:18 PM
To: user@hadoop.apache.org
Subject: Re: Is FileSystem thread-safe?

Hi John,

DistributedFileSystem is intended to be thread-safe, true to its name.

Metadata operations are handled by the NameNode server which synchronizes concurrent client
requests via locks (you can look at the FSNameSystem class).

Some discussion on the thread-safety aspects of HDFS:


On Sun, Mar 31, 2013 at 11:52 AM, Ted Yu <yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>
If you look at DistributedFileSystem source code, you would see that it calls the DFSClient
field member for most of the actions.
Requests to Namenode are then made through ClientProtocol.

An hdfs committer would be able to give you affirmative answer.

On Sun, Mar 31, 2013 at 11:27 AM, John Lilley <john.lilley@redpoint.net<mailto:john.lilley@redpoint.net>>
From: Ted Yu [mailto:yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>]
Subject: Re: Is FileSystem thread-safe?
>>FileSystem is an abstract class, what concrete class are you using (DistributedFileSystem,
etc) ?
Good point.  I am calling FileSystem.get(URI uri, Configuration conf) with an URI like "hdfs://server:port/..."
on a remote server, so I assume it is creating a DistributedFileSystem.  However I am not
finding any documentation discussing its thread-safety (or lack thereof), perhaps you can
point me to it?
Thanks, john

View raw message