Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0D3D10211 for ; Wed, 2 Oct 2013 04:54:42 +0000 (UTC) Received: (qmail 12102 invoked by uid 500); 2 Oct 2013 04:54:14 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 11832 invoked by uid 500); 2 Oct 2013 04:53:56 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 11794 invoked by uid 99); 2 Oct 2013 04:53:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Oct 2013 04:53:53 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [134.84.135.107] (HELO vs-a.tc.umn.edu) (134.84.135.107) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Oct 2013 04:53:48 +0000 Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) by vs-a.tc.umn.edu (UMN smtpd) with ESMTP for ; Tue, 1 Oct 2013 23:53:21 -0500 (CDT) X-Umn-Remote-Mta: [N] mail-wi0-f176.google.com [209.85.212.176] #+LO+TR X-Umn-Classification: local Received: by mail-wi0-f176.google.com with SMTP id cb5so6676148wib.3 for ; Tue, 01 Oct 2013 21:53:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=HIfGUOVm+ZMJNPIkqlZKuRUpwANpz78q6VTzhZqIXXg=; b=dDJ+IYwleqF1LkT6nNt/w+hO9M2aEgVm7jE5OJLUTRb5xs1KPSDqH1DjQqRjDikhNo 9t7ud4bO460tZFnt4vS4GSuIBqxq1VK2YHjic7c+/VEQi5EeR3Ukr6h0kH8DUPMQEGKg hRh7ci48mk4jyBs8ehPmqRP3xGmNmUSX7bVWOhjJ9nbRSH0DG79eOrNagAxOTT8Sm0fl NXCdqyKsf5zKqbHmFRFOOYxoN1BOo5Ph7CdGueY5xoW8hLMH96Fi9VriDVrgmElZ0wH/ e3YTfV4+pmafu+ppWJphinjg05tKWZSMGbmAaLg1MImfu6wgrdOQc6JuZlrO9WOT27N/ nGdQ== X-Gm-Message-State: ALoCoQlYyrFZlWa4GVod177pV96FyNQ05aFEBHw/cjztgWdALvCI4yIJdwFRxTjvACn2ZaADQLp5kVardoEquBAB4GEJR0fO1i+l3GOW3KXt8OYVEK6vUBur28Ckwfng3MZ4him/ZTxZwf0ZE+x2zD02VIcrpni0zw== X-Received: by 10.180.98.228 with SMTP id el4mr314459wib.4.1380689599927; Tue, 01 Oct 2013 21:53:19 -0700 (PDT) X-Received: by 10.180.98.228 with SMTP id el4mr314454wib.4.1380689599843; Tue, 01 Oct 2013 21:53:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.8.99 with HTTP; Tue, 1 Oct 2013 21:52:59 -0700 (PDT) From: Krishna Kumaar Natarajan Date: Tue, 1 Oct 2013 23:52:59 -0500 Message-ID: Subject: HDFS / Federated HDFS - Doubts To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d0445199f24d5a604e7bad937 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0445199f24d5a604e7bad937 Content-Type: text/plain; charset=ISO-8859-1 Hi All, While trying to understand federated HDFS in detail I had few doubts and listing them down for your help. 1. In case of *HDFS(without HDFS federation)*, the metadata or the data about the blocks belonging to the files in HDFS is maintained in the main memory of the name node or it is stored on permanent storage of the namenode and is brought in the main memory on demand basis ? [Krishna] Based on my understanding, I assume the entire metadata is in main memory which is an issue by itself. Please correct me if my understanding is wrong. 2. In case of* federated HDFS*, the metadata or the data about the blocks belonging to files in a particular namespace is maintained in the main memory of the namenode or it is stored on the permanent storage of the namenode and is brought in the main memory on demand basis ? 3. Are the metadata information stored in separate cluster nodes(block management layer separation) as discussed in Appendix B of this document ? https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf 4. I would like to know if the following proposals are already implemented in federated HDFS. ( http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability slide-17) - Separation of namespace and block management layers (same as qn.3) - Partial namespace in memory for further scalability - Move partial namespace from one namenode to another Thanks, Krishna --f46d0445199f24d5a604e7bad937 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi All,

While trying to understand federated HD= FS in detail I had few doubts and listing them down for your help.
  1. In case of HDFS(without HDFS federation), the me= tadata or the data about the blocks belonging to the files in HDFS is maint= ained in the main memory of the name node or it is stored on permanent stor= age of the namenode and is brought in the main memory on demand basis ?=A0[Krishna] Based on my understanding, I assume the entire m= etadata is in main memory which is an issue by itself. Please correct me if= my understanding is wrong.
  2. In case of federated HDFS<= /u>, the metadata or the data about the blocks belon= ging to files in a particular namespace is maintained in the main memory of= the namenode or it is stored on the permanent storage of the namenode and = is brought in the main memory on demand basis ?=A0
  3. Are the metadata information stored in separate clus= ter nodes(block management layer separation) as discussed in Appendix B of = this document ?https://issues.apache.org/jira/secure/attachm= ent/12453067/high-level-design.pdf
  4. I would like to know if the following proposals are = already implemented in federated HDFS. (http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for= -improved-efficiency-and-scalability=A0slide-17)
    • Separation of namespace and block managemen= t layers (same as qn.3)
    • Partial namespace= in memory for further scalability
    • Move partial namespace from one namenode to another<= /span>
Thanks,
Krishna
--f46d0445199f24d5a604e7bad937--