hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yong Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8286) Scaling out the namespace using KV store
Date Sat, 11 Jul 2015 03:20:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623205#comment-14623205

Yong Zhang commented on HDFS-8286:

If HDFS keep metadata in a persistent KV store, such as leveldb, is fsimage necessary?

> Scaling out the namespace using KV store
> ----------------------------------------
>                 Key: HDFS-8286
>                 URL: https://issues.apache.org/jira/browse/HDFS-8286
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: hdfs-kv-design.pdf
> Currently the NN keeps the namespace in the memory. To improve the scalability of the
namespace, users can scale up by using more RAM or scale out using Federation (i.e., statically
partitioning the namespace).
> We would like to remove the limitation of scaling the global namespace. Our vision is
that that HDFS should adopt a scalable underlying architecture that allows the global namespace
scales linearly.
> We propose to implement the HDFS namespace on top of a key-value (KV) store. Adopting
the KV store interfaces allows HDFS to leverage the capability of modern KV store and to become
much easier to scale. Going forward, the architecture allows distributing the namespace across
multiple machines, or  storing only the working set in the memory (HDFS-5389), both of which
allows  HDFS to manage billions of files using the commodity hardware available today.

This message was sent by Atlassian JIRA

View raw message