hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Mounting HDFS as local file system
Date Thu, 02 Dec 2010 13:10:18 GMT

On Dec 2, 2010, at 5:16 AM, Steve Loughran wrote:

> On 02/12/10 03:01, Mark Kerzner wrote:
>> Hi, guys,
>> I see that there is MountableHDFS<http://wiki.apache.org/hadoop/MountableHDFS>,
>> and I know that it works, but my questions are as follows:
>>    - How reliable is it for large storage?;
> Shouldn't be any worse than normal HDFS operations.
>>    - Is it not hiding the regular design questions - we are dealing with
>>    NameServers after all, but are trying to use it as a regular file system?
>>    - For example, HDFS is not optimized for many small files that get
>>    written and deleted, but a mounted system will lure one in this direction.
> Like you say, it's not a conventional posix fs, it hates small files, where other things
may be better.

I would comment that it's extremely reliable.  There's at least one slow memory leak in fuse-dfs
that I haven't been able to squash, and I typically remount things after a month or two of
*heavy* usage.

Across all the nodes in our cluster, we probably do a few billion HDFS operations per day
over FUSE.

View raw message