Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=from:to:references:subject:date:message-id:mime-version:
	content-type:content-transfer-encoding:x-mailer:x-mimeole:thread-index:in-reply-to;
	b=a6a8nTZ36Hd380mtdIJdyJPCiltVNWbKBFf+j5ztT1DaoKETgyGcg/s0E99zE6De
From: "Dhruba Borthakur" <dhruba@yahoo-inc.com>
To: <hadoop-user@lucene.apache.org>
References: <JM7BL3$B8B6C15D992941590FF1B080DE81FA0B@bol.com.br>
Subject: RE: HDFS and Small Files
Date: Fri, 3 Aug 2007 11:58:58 -0700
Message-ID: <02dd01c7d600$597ce590$639115ac@ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Thread-Index: AcfV2PcuvHuVrxK1T8uncygCREzUsQAJuBFQ
In-Reply-To: <JM7BL3$B8B6C15D992941590FF1B080DE81FA0B@bol.com.br>

How many small files do you have? What is the typical size of a file? What
are the file creation/deletion rates?

HDFS stores metadata information about each file in the NameNode's main
memory, so the number of files directly determines the size (CPU and memory)
required by the NameNode.

If you have a cluster with 10 million files, you might need to run the
NameNode on a machine that has 16 GB of ram.

Thanks
dhruba

-----Original Message-----
From: rlucindo [mailto:rlucindo@bol.com.br] 
Sent: Friday, August 03, 2007 7:15 AM
To: hadoop-user
Subject: HDFS and Small Files


I would like to know if anyone is using HDFS as a general purpose file
system (not for MapReduce). If so, how good is HDFS to handle lots of small
files?
I'm considering HDFS as an alternative to MogileFS, a big file system with
basically small files for a web application (the file system will store
html, images, videos, etc) where high availability is essential.
The documentation and wiki shows HDFS as a file system to support MapReduce
of big data volume, but not necessarily big files.

[]'s

Lucindo