hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements
Date Tue, 03 Mar 2015 18:46:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345482#comment-14345482

Colin Patrick McCabe commented on HDFS-7836:

bq. Generally we have some ways to use offheap, 1) directbuffer, it's not fit to our case.
2) Sun Unsafe. 3) Write native code and allocate memory ourselves. It seems we are going to
use #3?

I think #2 is the best.  As we talked about earlier, writing more JNI (#3) just adds more
platform dependencies that we don't want.  With regard to #1, allocating DirectByteBuffers
is very slow.

bq. If so, we still need to have an initial capacity for the hash table, otherwise there are
lots of rehash when the load factor reaches the threshold. How about we do the same thing
as what we currently do in java blockmap, using 2% of totoal memory? Then we don't need the
load factor and assume the table rows number are enough?

That's an interesting idea.  Keep in mind, though, that this hash table will be off-heap.
 So if we size the hash table based on the JVM heap size, it would be weird.  Honestly I think
having a setting for this is easiest.  I also suspect that growing the off-heap hash table
will be quicker than growing an on-heap hash table, since it won't trigger full GCs during

> BlockManager Scalability Improvements
> -------------------------------------
>                 Key: HDFS-7836
>                 URL: https://issues.apache.org/jira/browse/HDFS-7836
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Charles Lamb
>            Assignee: Charles Lamb
>         Attachments: BlockManagerScalabilityImprovementsDesign.pdf
> Improvements to BlockManager scalability.

This message was sent by Atlassian JIRA

View raw message