hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7240) Object store in HDFS
Date Tue, 30 Jun 2015 20:47:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609037#comment-14609037

Jitendra Nath Pandey commented on HDFS-7240:

[~john.jian.fang], thanks for review and comments
1) There is some work going on to support yarn for s3 (HADOOP-11262) by [~thodemoor] and [~steve_l]].
We hope we can leverage that to support mapreduce use case.
2) This will be part of hdfs natively, unlike hbase which is a separate service. There is
only one additional daemon needed (storage container manager) only if ozone is deployed. Hdfs
already supports multiple namespaces and blockpools, we will leverage that work. This work
will not impact hdfs manageability in anyway. There will be some additional code in datanode
to support storage containers. In future, we hope to use containers to store hdfs blocks as
well for hdfs block-space scalability.
3) We will partition objects using hash partitioning and range partitioning. The document
already talks about hash partitioning, I will add more details for range partition support.
The small objects will also be stored in the container. In the document I mentioned leveldbjni,
but we are also looking at rocksDB for container implementation to store objects in the container.
4) The containers will be replicated and kept consistent using data pipeline, similar to hdfs
blocks. The document talks about it at a high level. We will update more details in the related

> Object store in HDFS
> --------------------
>                 Key: HDFS-7240
>                 URL: https://issues.apache.org/jira/browse/HDFS-7240
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: Ozone-architecture-v1.pdf
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a generic storage
layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the
storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode storage, but
independent of namespace metadata.
> I will soon update with a detailed design document.

This message was sent by Atlassian JIRA

View raw message