hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5477) Block manager as a service
Date Mon, 13 Jan 2014 18:30:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869798#comment-13869798
] 

Colin Patrick McCabe commented on HDFS-5477:
--------------------------------------------

Hi Daryn,

This seems like a great direction for HDFS to go in.  Just a few comments.

You list scalability as a primary concern.  However, even if we separate the BM from the namespace
management, a cluster with a large number of blocks will still have a giant BM heap (if I
understand correctly).  So perhaps what we need is the ability to have multiple block manager
daemons?

It seems like there will be a lot of messages that will necessarily flow between the namespace
daemon and the block management daemon(s).  What IPC mechanism are you considering?  TCP socket?
 UNIX domain socket?  Shared memory?  Shared memory would clearly be the highest performance,
and perhaps we should consider that.

Is there an upstream svn branch for this yet?

> Block manager as a service
> --------------------------
>
>                 Key: HDFS-5477
>                 URL: https://issues.apache.org/jira/browse/HDFS-5477
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf, Standalone BM.pdf
>
>
> The block manager needs to evolve towards having the ability to run as a standalone service
to improve NN vertical and horizontal scalability.  The goal is reducing the memory footprint
of the NN proper to support larger namespaces, and improve overall performance by decoupling
the block manager from the namespace and its lock.  Ideally, a distinct BM will be transparent
to clients and DNs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message