Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 2 Jan 2014 22:03:50 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12687037.1388695566668.33980.1388700230780@arcas>
In-Reply-To: <JIRA.12687037.1388695566668@arcas>
References: <JIRA.12687037.1388695566668@arcas>
Subject: [jira] [Commented] (HDFS-5711) Removing memory limitation of the
 Namenode by persisting Block - Block location mappings to disk.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HDFS-5711?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13860=
846#comment-13860846 ]=20

Sanjay Radia commented on HDFS-5711:
------------------------------------

bq. We also intend to use LevelDB to persist metadata, and plan to provide =
a complete solution, by not just persisting the Namespace information but a=
lso the Blocks Map onto secondary storage.=20

The namespace layer and block layer have been separated reasonably well as =
part of the federation work. Hence it makes sense to keep the persistence o=
f the namespace and the persistence of the block map as two separate jiras.=
=20
Since there  is already a Jira, HDFS-5389,  focusing on storing  a portion =
(the working set ) of the  namespace in memory, this jira should focus on t=
he storing the block mapping on disk (as stated in the title).


> Removing memory limitation of the Namenode by persisting Block - Block lo=
cation mappings to disk.
> -------------------------------------------------------------------------=
------------------------
>
>                 Key: HDFS-5711
>                 URL: https://issues.apache.org/jira/browse/HDFS-5711
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Rohan Pasalkar
>
> This jira is to track changes to be made to remove HDFS name-node memory =
limitation to hold block - block location mappings.
> It is a known fact that the single Name-node architecture of HDFS has sca=
lability limits. The HDFS federation project alleviates this problem by usi=
ng horizontal scaling. This helps increase the throughput of metadata opera=
tion and also the amount of data that can be stored in a Hadoop cluster.
> The Name-node stores all the filesystem metadata in memory (even in the f=
ederated architecture), the
> Name-node design can be enhanced by persisting part of the metadata onto =
secondary storage and retaining=20
> the popular or recently accessed metadata information in main memory. Thi=
s design can benefit a HDFS deployment
> which doesn't use federation but needs to store a large number of files o=
r large number of blocks. Lin Xiao from Hortonworks attempted a similar
> project [1] in the Summer of 2013. They used LevelDB to persist the Names=
pace information (i.e file and directory inode information).
> A patch with this change is yet to be submitted to code base. We also int=
end to use LevelDB to persist metadata, and plan to=20
> provide a complete solution, by not just persisting  the Namespace inform=
ation but also the Blocks Map onto secondary storage.=20
> We did implement the basic prototype which stores the block-block locatio=
n mapping metadata to the persistent key-value store i.e. levelDB. Prototyp=
e also maintains the in-memory cache of the recently used block-block locat=
ion mappings metadata.=20
> References:
> [1] Lin Xiao, Hortonworks, Removing Name-node=E2=80=99s memory limitation=
, http://www.slideshare.net/ydn/hadoop-meetup-hug-august-2013-removing-the-=
namenodes-memory-limitation


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)