Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of pwyckoff@facebook.com
 designates 204.15.23.140 as permitted sender)
From: Pete Wyckoff <pwyckoff@facebook.com>
To: "core-user@hadoop.apache.org" <core-user@hadoop.apache.org>
Date: Mon, 3 Nov 2008 11:26:17 -0800
Subject: Re: Status FUSE-Support of HDFS
Thread-Topic: Status FUSE-Support of HDFS
Thread-Index: Ack9o4+durLkyrgGSRSdhoZYYkLuqgARnoE/
Message-ID: <C5349057.33478%pwyckoff@facebook.com>
In-Reply-To: <490ED9C8.4070103@apache.org>
Accept-Language: en, en-US
Content-Language: en
acceptlanguage: en, en-US
Content-Type: multipart/alternative;
	boundary="_000_C534905733478pwyckofffacebookcom_"
MIME-Version: 1.0

--_000_C534905733478pwyckofffacebookcom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


+1 but since hadoop deals well with such directories currently, fuse-dfs wi=
ll basically lock up on such directories - this is because ls -color=3Dblah=
 causes a stat on every file in a directory.  There is a JIRA open for this=
 but it is a pretty rare case although it has happened to me at facebook.

-- pete


>It's good for a portable application to
keep the #of files/directory low by having two levels of directory for
storing files -just use a hash operation to determine which dir to store
a specific file in.


On 11/3/08 4:00 AM, "Steve Loughran" <stevel@apache.org> wrote:

Pete Wyckoff wrote:
> It has come a long way since 0.18 and facebook keeps our (0.17) dfs mount=
ed via fuse and uses that for some operations.
>
> There have recently been some problems with fuse-dfs when used in a multi=
threaded environment, but those have been fixed in 0.18.2 and 0.19. (do not=
 use 0.18 or 0.18.1)
>
> The current (known) issues are:

>   2. When directories have 10s of thousands of files, performance can be =
very poor.

I've known other filesystems to top out at 64k-1 files per directory,
even if they don't slow down. It's good for a portable application to
keep the #of files/directory low by having two levels of directory for
storing files -just use a hash operation to determine which dir to store
a specific file in.


--_000_C534905733478pwyckofffacebookcom_--