hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pete Wyckoff <pwyck...@facebook.com>
Subject Re: Status FUSE-Support of HDFS
Date Mon, 03 Nov 2008 19:26:17 GMT

+1 but since hadoop deals well with such directories currently, fuse-dfs will basically lock
up on such directories - this is because ls -color=blah causes a stat on every file in a directory.
 There is a JIRA open for this but it is a pretty rare case although it has happened to me
at facebook.

-- pete


>It's good for a portable application to
keep the #of files/directory low by having two levels of directory for
storing files -just use a hash operation to determine which dir to store
a specific file in.


On 11/3/08 4:00 AM, "Steve Loughran" <stevel@apache.org> wrote:

Pete Wyckoff wrote:
> It has come a long way since 0.18 and facebook keeps our (0.17) dfs mounted via fuse
and uses that for some operations.
>
> There have recently been some problems with fuse-dfs when used in a multithreaded environment,
but those have been fixed in 0.18.2 and 0.19. (do not use 0.18 or 0.18.1)
>
> The current (known) issues are:

>   2. When directories have 10s of thousands of files, performance can be very poor.

I've known other filesystems to top out at 64k-1 files per directory,
even if they don't slow down. It's good for a portable application to
keep the #of files/directory low by having two levels of directory for
storing files -just use a hash operation to determine which dir to store
a specific file in.



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message