hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Lamb (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements
Date Thu, 26 Feb 2015 23:12:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339385#comment-14339385

Charles Lamb commented on HDFS-7836:

bq. it would be useful to see some perf comparison before we add that complexity.

We definitely plan on getting some baseline measurements and sharing them. We definitely want
to know what the before and after effects are of any changes. As an aside, I worked on a case
where we had to increase the RPC limit to 192MB in order to get block reports handled correctly
so I know this type of deployments are out there.

bq. I'll see if I can clean up and post what I used on HDFS-7847.

That would be much appreciated. I'm starting to look at HDFS-7847 (subtask of this Jira) and
maybe that could come into play somehow.

bq. These two sound contradictory.

Yes, they do, but aren't meant to be. The first level would be to do concurrent processing
under the FSN lock. That would at least get some parallelism. The second step would be to
make a more "lockless blocksMap" which wouldn't require the big FSN lock to be held.

BTW, since you have edit privs, if you want to get rid of my reduntent reply to you above
that would be great. My browser suckered me into hitting "Add" twice. Thanks.

> BlockManager Scalability Improvements
> -------------------------------------
>                 Key: HDFS-7836
>                 URL: https://issues.apache.org/jira/browse/HDFS-7836
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Charles Lamb
>            Assignee: Charles Lamb
>         Attachments: BlockManagerScalabilityImprovementsDesign.pdf
> Improvements to BlockManager scalability.

This message was sent by Atlassian JIRA

View raw message