Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 26 Feb 2015 23:12:06 +0000 (UTC)
From: "Charles Lamb (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12777364.1424824988000.28690.1424992326397@Atlassian.JIRA>
In-Reply-To: <JIRA.12777364.1424824988000@Atlassian.JIRA>
References: <JIRA.12777364.1424824988000@Atlassian.JIRA>
 <JIRA.12777364.1424824988562@arcas>
Subject: [jira] [Commented] (HDFS-7836) BlockManager Scalability
 Improvements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339385#comment-14339385 ] 

Charles Lamb commented on HDFS-7836:
------------------------------------

bq. it would be useful to see some perf comparison before we add that complexity.

We definitely plan on getting some baseline measurements and sharing them. We definitely want to know what the before and after effects are of any changes. As an aside, I worked on a case where we had to increase the RPC limit to 192MB in order to get block reports handled correctly so I know this type of deployments are out there.

bq. I'll see if I can clean up and post what I used on HDFS-7847.

That would be much appreciated. I'm starting to look at HDFS-7847 (subtask of this Jira) and maybe that could come into play somehow.

bq. These two sound contradictory.

Yes, they do, but aren't meant to be. The first level would be to do concurrent processing under the FSN lock. That would at least get some parallelism. The second step would be to make a more "lockless blocksMap" which wouldn't require the big FSN lock to be held.

BTW, since you have edit privs, if you want to get rid of my reduntent reply to you above that would be great. My browser suckered me into hitting "Add" twice. Thanks.

> BlockManager Scalability Improvements
> -------------------------------------
>
>                 Key: HDFS-7836
>                 URL: https://issues.apache.org/jira/browse/HDFS-7836
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Charles Lamb
>            Assignee: Charles Lamb
>         Attachments: BlockManagerScalabilityImprovementsDesign.pdf
>
>
> Improvements to BlockManager scalability.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)