hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-13074) Ozone File System
Date Sat, 27 Jan 2018 04:40:01 GMT
Jitendra Nath Pandey created HDFS-13074:

             Summary: Ozone File System
                 Key: HDFS-13074
                 URL: https://issues.apache.org/jira/browse/HDFS-13074
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: hdfs
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey

This jira splits out the Ozone's Key Value namespace out of HDFS-7240, leaving that jira to
focus on the block layer Hadoop Distributed Storage Layer (HDSL).  HDFS-10419 focuses on
the traditional hierarchical namespace/NN on top of HDFS while this jira focuses on a flat
Key-Value namespace call ed Ozone FS on top HDSL. 

[~owen.omalley] suggested the split in HDFS-7240 in this comment.

Ozone provides two APIs:
 * A Hadoop compatible FS (Haddop FileSystem and Hadoop FileContext), on top of KV API.

Ozone FS serves the following purpose
 * It helps test the new storage layer (HDSL)
 * It can be directly used by applications (such. Hive, Spark)  that are ready for cloud's
native storage systems that use a KV namespace such as S3, Azure ADLS.

Ozone's namespace server scales well because of following reasons:
 * It keeps only the working set of metadata in memory - this scales the Key-Value namespace.
 * Ozone does not need to keep the equivalent of the block-map; instead the corresponding
container-to-location mapping resides in in the SCM - this leaves additional free space for
caching the metadata. This benefit is also available to NN that adapts to the new block-storage
container layer as explained in Evolving NN using new block container layer attached in HDFS-10419.
 * It does not have a single global lock - this will scale well against a large number of
concurrent clients/rpcs. 
 * It can be partitioned/sharded much more easily because it is a flat namespace. Indeed a
natural partitioning level is the bucket. Partitioning would further scale the number of clients/rpcs
and also the Key-value namespace.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message