hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements
Date Tue, 03 Mar 2015 08:38:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344735#comment-14344735

Yi Liu commented on HDFS-7836:

Good proposal, Colin and Charles. 

Reduce lock contention for the FSNamesystem lock
Enable concurrent processing of block reports

Maybe at the first phase, we can separate locks for block reports from FSN lock. Even so,
It's a good improvement and include bulk of work. 
Then we can do further improvement to have multiple locks (stripes) for block reports . 

Use offheap to reduce the Java heap size of the NameNode
Generally we have some ways to use offheap, 1) directbuffer, it's not fit to our case. 2)
Sun Unsafe. 3) Write native code and allocate memory ourselves. It  seems we are going to
use #3?

We probably want to support dynamically growing the hash table, to avoid putting too much
of a
burden on the administrator to configure the size

If so, we still need to have an initial capacity for the hash table, otherwise there are lots
of rehash when the load factor reaches the threshold.
How about we do the same thing as what we currently do in java blockmap, using 2% of totoal
memory? Then we don't need the load factor and assume the table rows number are enough?

> BlockManager Scalability Improvements
> -------------------------------------
>                 Key: HDFS-7836
>                 URL: https://issues.apache.org/jira/browse/HDFS-7836
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Charles Lamb
>            Assignee: Charles Lamb
>         Attachments: BlockManagerScalabilityImprovementsDesign.pdf
> Improvements to BlockManager scalability.

This message was sent by Atlassian JIRA

View raw message