hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <ev...@yahoo-inc.com>
Subject Re: [Proposal] Pluggable Namespace
Date Fri, 04 Oct 2013 15:31:27 GMT
I think it would be awesome to make this happen.  I am +1 for the general
direction. I am by no means an HDFS expert but I would like to see the
patches you have made.  If you could file a JIRA that points to the
modifications on github or in patches and any design work you have
explaining it (including what you have below) that would give us a place
to review it and discuss it.


On 10/3/13 2:17 PM, "Milind Bhandarkar" <mbhandarkar@gopivotal.com> wrote:

>Hi All,
>Exec Summary: For the last couple of months, we, at Pivotal, along with a
>couple of folks in the community have been working on making Namespace
>implementation in the namenode pluggable. We have demonstrated that it can
>be done without major surgery on the namenode, and does not have
>performance impact. We would like to contribute it back to Apache if there
>is sufficient interest. Please let us know if you are interested, and we
>will create a Jira and update the patch for in-progress work.
>In a Hadoop cluster, Namenode roughly has following main responsibilities.
>€ Catering to RPC calls from clients.
>€ Managing the HDFS namespace tree.
>€ Managing block report, heartbeat and other communication from data
>For Hadoop clusters having large number of files and large number of
>name node gets bottlenecked. Mainly for two reasons
>€ All the information is kept in name node¹s main memory.
>€ Namenode has to cater to all the request from clients / data nodes.
>€ And also perform some operations for backup and check pointing node.
>A possible solution is to add more main memory but there are certain
>with this approach
>€ Namnenode being Java application, garbage collection cycles execute
>periodically to reclaim unreferenced heap space. When the heap space grows
>very large, despite of GC policy  chosen, application stalls during the GC
>activity. This creates a bunch of issues since DNs and  clients may
>perceive this stall as NN crash.
>€ There will always be a practical limit on how much physical memory a
>single machine can accommodate.
>Proposed Solution:
>Out of the three responsibilities listed above, we can refactor namespace
>management from the namenode codebase in such a way that there is
>to implement and plug other name systems other than existing in-process
>memory-based name system. Particularly a name system backed by a
>distributed key-value store will significantly reduce namenode memory
>requirement.To achieve this, a new generic interface will be introduced
>[Let¹s call it AbstractNameSystem] which defines set of operations using
>which we perform the namespace management. Namenode code that used to
>manipulate some java objects maintained in namenode¹s heap will now
>on this interface. There will be provision for others to extend this
>interface and plug their own NameSystem implementation.
>To get started, we have implemented the same memory-based namespace
>implementation in a remote process, outside of the namenode JVM. In
>addition, work is undergoing to implement the namesystem using HBase.
>Details of Changes:
>Created new class called AbstractNamesystem, existing FSNamesystem is a
>subclass of this class. Some code from FSNamesystem has been moved to its
>parent. Created a Factory class to create object of NS management
>class.Factory refers to newly added config properties to support pluggable
>name space management class. Added unit tests for Factory. Replaced
>constructors with factory calls, this is  because the namesystem instances
>should now be created based on configuration. Added new config properties
>to support pluggable name space management class. This property will
>which Namesystem class will be instantiated by the factory. This change is
>also reflected in some DFS related webapps [JSP files] where namesystem
>instance is used to obtain DFS health and other stats.
>These changes aim to make the namesystem pluggable without changing high
>level interfaces, this is particularly tricky since memory-based name
>system functionality is currently baked into these interfaces, and
>goal is to make the high level interface free from memory-based name
>Consideration for Upgrade and Rollback:
>Current memory based implementation already has code to read from and
>to fsimage , we will have to make them publicly accessible which will
>enable us to upgrade an existing cluster from FSNamespace to newly added
>name system in future version.
>a. Upgrades: By making use of existing Loader class for reading fsimage we
>can write some code load this image into the future name system
>b. Rollback: Are even simpler, we can preserve the old fsimage and start
>the cluster with that image by configuring the cluster to use current file
>system based name system.
>Future work
>Current HDFS design is such that FSNameSystem is baked into even high
>interfaces, this is a major hurdle in cleanly implementing pluggable name
>systems. We aim to propose a change in such interfaces into which
>FSNameSystem is tightly coupled.
>- Milind
>Milind Bhandarkar
>Chief Scientist

View raw message