Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of markkerzner@gmail.com
 designates 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=oolWn9ClGudODPfoLQ2ijwDNCwftD/FbJyt+Al1J4LHMK6GIzgK+3vPcahX1dwsGbQ
         Eh/nmGmmfSQrUclXyeRDtfCh5Hst6x0COtjaewB7Fo9LSFK22srsQ1KUVQya4agIJ7c/
         sI6+JdAibaLXwm5yjTMtkihuPU12ydRrh6jUI=
MIME-Version: 1.0
In-Reply-To: <35DDD480-AA70-4FE6-9C98-D4CE1A9F0F77@cse.unl.edu>
References: <AANLkTinv2Pf+0CmubYigaiKkXkpCkm9_fmZoPO=yyt9r@mail.gmail.com>
	<4CF78006.9060407@apache.org>
	<35DDD480-AA70-4FE6-9C98-D4CE1A9F0F77@cse.unl.edu>
Date: Thu, 2 Dec 2010 08:52:07 -0600
Message-ID: <AANLkTikQ2sfnt0By1BaUPvLkOVzjOfAM+kOjxb7oR28t@mail.gmail.com>
Subject: Re: Mounting HDFS as local file system
From: Mark Kerzner <markkerzner@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a531db481c04966e9018

--20cf3054a531db481c04966e9018
Content-Type: text/plain; charset=ISO-8859-1

Thank you, Brian.

I found your paper "Using Hadoop as grid storage," and it was very useful.

One thing I did not understand in it is your file usage pattern - do you
deal with small or large files, and do you delete them often enough? My
question was, in part, can you use HDFS as a regular file system with
frequent file deletes? Does it not become fragmented and unreliable?

Thank you,
Mark

On Thu, Dec 2, 2010 at 7:10 AM, Brian Bockelman <bbockelm@cse.unl.edu>wrote:

>
> On Dec 2, 2010, at 5:16 AM, Steve Loughran wrote:
>
> > On 02/12/10 03:01, Mark Kerzner wrote:
> >> Hi, guys,
> >>
> >> I see that there is MountableHDFS<
> http://wiki.apache.org/hadoop/MountableHDFS>,
> >> and I know that it works, but my questions are as follows:
> >>
> >>    - How reliable is it for large storage?;
> >
> > Shouldn't be any worse than normal HDFS operations.
> >
> >>    - Is it not hiding the regular design questions - we are dealing with
> >>    NameServers after all, but are trying to use it as a regular file
> system?
> >>    - For example, HDFS is not optimized for many small files that get
> >>    written and deleted, but a mounted system will lure one in this
> direction.
> >
> > Like you say, it's not a conventional posix fs, it hates small files,
> where other things may be better.
>
> I would comment that it's extremely reliable.  There's at least one slow
> memory leak in fuse-dfs that I haven't been able to squash, and I typically
> remount things after a month or two of *heavy* usage.
>
> Across all the nodes in our cluster, we probably do a few billion HDFS
> operations per day over FUSE.
>
> Brian

--20cf3054a531db481c04966e9018--