Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 76210 invoked from network); 13 Feb 2009 22:16:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Feb 2009 22:16:49 -0000 Received: (qmail 49637 invoked by uid 500); 13 Feb 2009 22:16:48 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 49606 invoked by uid 500); 13 Feb 2009 22:16:48 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 49597 invoked by uid 99); 13 Feb 2009 22:16:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2009 14:16:48 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [192.87.106.226] (HELO aurora.apache.org) (192.87.106.226) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2009 22:16:47 +0000 Received: from aurora.apache.org (localhost [127.0.0.1]) by aurora.apache.org (8.13.8+Sun/8.13.8) with ESMTP id n1DMGOZ8002170 for ; Fri, 13 Feb 2009 22:16:26 GMT Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki To: core-commits@hadoop.apache.org Date: Fri, 13 Feb 2009 22:16:24 -0000 Message-ID: <20090213221624.2088.43849@aurora.apache.org> Subject: [Hadoop Wiki] Update of "HdfsFutures" by SanjayRadia X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by SanjayRadia: http://wiki.apache.org/hadoop/HdfsFutures ------------------------------------------------------------------------------ === Scaling Namespace (i.e. number of files/dirs) === + Since the name node stores block and name objects in memory, the size of the name space (and hence the number of files) is limited by amount of heap memory. Currently a 14GB heap (ie 16GB machine) allows 60 million block and name objects. Hence + if one has 2 blocks per file, then one is limited to 20 million files. Partition/distribute Name node (will also help performance) [[BR]]Several Options: