hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8392) DataNode support for multiple datasets
Date Thu, 14 May 2015 17:30:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544064#comment-14544064
] 

Arpit Agarwal commented on HDFS-8392:
-------------------------------------

FsDatasetSpi is geared towards storing and retrieving files. In the object store we want to
be able to store and retrieve metadata containers and data containers. Files may not be the
best abstraction for these containers. For these we'll introduce a StorageContainerDataset.
We don't foresee a third dataset type right now.

The DataNode already supports multiple block pools per storage volume and most of the difficult
work was done as part of the federation feature. It is relatively straightforward to extend
it to support the notion of a dataset per block pool. So in a cluster running non-federated
HDFS and Object store services, the DataNodes would have two blockpools and two datasets,
each servicing one block pool.

Hope that's a little clearer. I intend to post a patch next week.

> DataNode support for multiple datasets
> --------------------------------------
>
>                 Key: HDFS-8392
>                 URL: https://issues.apache.org/jira/browse/HDFS-8392
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>
> For HDFS-7240 we would like to share available DataNode storage across HDFS blocks and
Ozone objects.
> The DataNode already supports sharing available storage across multiple block pool IDs
for the federation feature. However all federated block pools use the same dataset implementation
i.e. {{FsDatasetImpl}}.
> We can extend the DataNode to support multiple dataset implementations so the same storage
space can be shared across one or more HDFS block pools and one or more Ozone block pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message