hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Thomas <tho...@hep.caltech.edu>
Subject Re: Mounting HDFS as local file system
Date Thu, 02 Dec 2010 15:25:02 GMT
On 12/02/2010 05:10 AM, Brian Bockelman wrote:
> On Dec 2, 2010, at 5:16 AM, Steve Loughran wrote:
>> On 02/12/10 03:01, Mark Kerzner wrote:
>>> Hi, guys,
>>> I see that there is MountableHDFS<http://wiki.apache.org/hadoop/MountableHDFS>,
>>> and I know that it works, but my questions are as follows:
>>>    - How reliable is it for large storage?;
>> Shouldn't be any worse than normal HDFS operations.
>>>    - Is it not hiding the regular design questions - we are dealing with
>>>    NameServers after all, but are trying to use it as a regular file system?
>>>    - For example, HDFS is not optimized for many small files that get
>>>    written and deleted, but a mounted system will lure one in this direction.
>> Like you say, it's not a conventional posix fs, it hates small files, where other
things may be better.
> I would comment that it's extremely reliable.  There's at least one slow memory leak
in fuse-dfs that I haven't been able to squash, and I typically remount things after a month
or two of *heavy* usage.

Using the automounter with fuse-dfs has helped us a lot.  After 5
minutes of no activity, the fuse-dfs process goes away and the memory
leak is cleaned up automatically.  We only see the problem when there is
constant HDFS usage for days at a time, which unfortunately, has been
the rule rather than the exception lately.


> Across all the nodes in our cluster, we probably do a few billion HDFS operations per
day over FUSE.

View raw message