hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sheldon <dave-had...@earth.li>
Subject Sensible limits on file numbers on HDFS
Date Mon, 05 Nov 2007 15:13:33 GMT

I'm considering using HDFS for storing a large number of files. 

We will be storing a couple of million files, each 5-30 megabytes in
size. There will be few writes, and many reads of these files.

Am I right to be worried about the memory requirements on the name
servers? What sort of requirements would this have? Would mogilefs be a
better fit for this sort of load, or maybe HBase? or is there a better
option that I haven't considered?

I'm sure this should be in the FAQ, but all I have found is vague
"designed for few, large files", and similar warnings.

Thanks for your help.


   Dijkstra probably hates me -- Linus Torvalds, in kernel/sched.c

View raw message