hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7240) Object store in HDFS
Date Thu, 02 Nov 2017 02:11:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235080#comment-16235080

Konstantin Shvachko commented on HDFS-7240:

??I hope this addresses your concerns.??

I don't think _that_ addressed any of my concerns.
* Ozone by itself does not solve any of HDFS problems. It uses HDFS-agnostic S3-like API,
and I cannot use it on my clusters.
Unless I can convince thousands of my users to rewrite their thousands of applications, along
with the existing computational frameworks: YARN, Hive, Pig, Spark, ......  created over the
past 10 years.
* I was talking about futuristic architecture, when you start using Ozone for block management,
and rewrite NameNode to store its namespace in LevelDB. If this is still your plan. I agree
this architecture solves the objects-count problem. But it does not solve the problem of scaling
RPC requests, which is more important to me than the # of objects, since you still cannot
grow the cluster beyond the single-NameNode's-RPC-processing capability.

> Object store in HDFS
> --------------------
>                 Key: HDFS-7240
>                 URL: https://issues.apache.org/jira/browse/HDFS-7240
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Major
>         Attachments: HDFS-7240.001.patch, HDFS-7240.002.patch, HDFS-7240.003.patch, HDFS-7240.003.patch,
HDFS-7240.004.patch, Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, ozone_user_v0.pdf
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a generic storage
layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the
storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode storage, but
independent of namespace metadata.
> I will soon update with a detailed design document.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message