hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7240) Object store in HDFS
Date Thu, 09 Apr 2015 05:34:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486738#comment-14486738

Lars Hofhansl commented on HDFS-7240:

Awesome stuff. We (Salesforce) have a need for this.

I think these will lead to immediate management problems: 
* Object Size : 5G
* Number of buckets system-­wide : 10 million  
* Number of objects per bucket: 1 million  
* Number of buckets per storage volume : 1000  

We have a large number of tenant (many times more than 1000). Some of the tenants will be
very large (storing many times more than 1m objects). Of course there are simple workarounds
for that, such as including a tenant id in the volume name and a bucket name in our internal
blob ids. Are these technical limits?

I don't think that we're the only ones who will to store a large amount of objects (more than
1m) and the bucket management would get into the way, rather than help.

> Object store in HDFS
> --------------------
>                 Key: HDFS-7240
>                 URL: https://issues.apache.org/jira/browse/HDFS-7240
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: Ozone-architecture-v1.pdf
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a generic storage
layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the
storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode storage, but
independent of namespace metadata.
> I will soon update with a detailed design document.

This message was sent by Atlassian JIRA

View raw message