hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10467) Router-based HDFS federation
Date Fri, 27 May 2016 19:58:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304668#comment-15304668

Inigo Goiri commented on HDFS-10467:

This approach should support the scenarios in the mail thread. Right now we support the most
typical operations of the RPC interface and the basic ones for REST. We don't do proxying
of the requests to the DNs as HttpFs does but at some point we might; for now we just point
HttpFs to our Routers. Instead of extending HttpFs we went into mimicing the NN to provide
the full interface (i.e., RPC) and mimic the image of a big large NN.

The main difference with the approach proposed in the mail thread is the addition of the State
Store as a centralized storage for the federation state. This mimics the architecture from
YARN federation in YARN-2915.

If the document is not enough, I can provide a patch with the full approach (including Router,
State Store and a simple cluster rebalancer). I think later on, we should split it into multiple

> Router-based HDFS federation
> ----------------------------
>                 Key: HDFS-10467
>                 URL: https://issues.apache.org/jira/browse/HDFS-10467
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.7.2
>            Reporter: Inigo Goiri
>         Attachments: HDFS Router Federation.pdf
> Add a Router to provide a federated view of multiple HDFS clusters.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message