hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10467) Router-based HDFS federation
Date Thu, 22 Sep 2016 01:58:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511831#comment-15511831

Subru Krishnan commented on HDFS-10467:

Thanks [~jakace] and [~goiri] for the refactored patch. I made a quick pass in the context
of HADOOP-13378 and I think that we can represent the YARN {{FederationStateStore}} using
the generic {{StateStoreDriver}} you guys have defined. Personally I prefer the push mechanism
we have in YARN-3671 as it's much simpler than the pull mechanism proposed here though I do
agree both achieve the same result.

A couple of comments based on my quick scan:
  * We should add versioning to the generic {{StateStoreDriver}}. Refer [FederationStateStore|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/FederationStateStore.java].
  * Use _jcache_ instead of writing custom key-caches based on _ConcurrentHashMaps_. In fact,
I feel we can refactor the (jcache-based) cache in [FederationStateStoreFacade|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java]
and use it across both efforts.
  * Lastly I would suggest using [Curator|http://curator.apache.org/] for *ZooKeeper* implementation
as we have moved to it in YARN (YARN-4438 and follow up work).

> Router-based HDFS federation
> ----------------------------
>                 Key: HDFS-10467
>                 URL: https://issues.apache.org/jira/browse/HDFS-10467
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.7.2
>            Reporter: Inigo Goiri
>            Assignee: Inigo Goiri
>         Attachments: HDFS Router Federation.pdf, HDFS-10467.PoC.001.patch, HDFS-10467.PoC.patch,
> Add a Router to provide a federated view of multiple HDFS clusters.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message