hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-9806) Allow HDFS block replicas to be provided by an external storage system
Date Tue, 08 Aug 2017 17:15:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118654#comment-16118654
] 

Sean Mackrory edited comment on HDFS-9806 at 8/8/17 5:14 PM:
-------------------------------------------------------------

Could someone more familiar with the design here comment on any anticipated impact to NameNode
scalability? If this feature is included but a user chooses not to use it (i.e. there are
no storages of type PROVIDED but the capability is there) - is there any impact to memory
consumed by the NameNode? I've read the design doc and some of the patches - I *think* things
are good but wanting to be sure by someone more familiar with the implementation and who might
have even tested that already...


was (Author: mackrorysd):
Could someone more familiar with the design here comment on any anticipated impact to NameNode
scalability? If this feature is included but a user chooses not to use it (i.e. there are
no storages of type PROVIDED) - is there any impact to memory consumed by the NameNode? I've
read the design doc and some of the patches - I *think* things are good but wanting to be
sure by someone more familiar with the implementation and who might have even tested that
already...

> Allow HDFS block replicas to be provided by an external storage system
> ----------------------------------------------------------------------
>
>                 Key: HDFS-9806
>                 URL: https://issues.apache.org/jira/browse/HDFS-9806
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Chris Douglas
>         Attachments: HDFS-9806-design.001.pdf, HDFS-9806-design.002.pdf
>
>
> In addition to heterogeneous media, many applications work with heterogeneous storage
systems. The guarantees and semantics provided by these systems are often similar, but not
identical to those of [HDFS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html].
Any client accessing multiple storage systems is responsible for reasoning about each system
independently, and must propagate/and renew credentials for each store.
> Remote stores could be mounted under HDFS. Block locations could be mapped to immutable
file regions, opaque IDs, or other tokens that represent a consistent view of the data. While
correctness for arbitrary operations requires careful coordination between stores, in practice
we can provide workable semantics with weaker guarantees.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message