hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Reason to store 64 block file in a sub directory?
Date Mon, 11 Oct 2010 20:46:24 GMT
If I recall correctly, ext3 has O(n) performance for lookup of a
directory entry. So, having thousands of files in a directory is bad
for performance. Additionally, there's a max of 31998 files in a
directory, so you have to split into subdirs eventually.


On Mon, Oct 11, 2010 at 8:15 PM, Thanh Do <thanhdo@cs.wisc.edu> wrote:
> Hi all,
> can anyone explain to me while do HDFS has the policy
>  to store 64 block files in a single sub directory?
> and if the number of block files increase,
> it just simply creates another subdir and put the block files there.
> Thanks
> Thanh

Todd Lipcon
Software Engineer, Cloudera

View raw message