hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luo Ning (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HBASE-823) Concurrent "open mapfile reader" limit
Date Sun, 08 Feb 2009 06:18:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671562#action_12671562
] 

ln@webcate.net edited comment on HBASE-823 at 2/7/09 10:18 PM:
---------------------------------------------------------

i think we need archiving 2 goals for the 'open reader limit':
1. fit xceiverCount limitation of hadoop, see HBASE-24
2. control memory usage of mapfile indexes.

we may got 2 by control 'concurrent open readers', but it is not efficient. because mapfile
index size is very different. for regionserver stable, we should make sure 'open file limit'
* 'max  mapfile index size' < 'memory limit of regionserver'. this mean set the 'open file
limit' low enough.

so besides 'open reader limit', there should a 'max index size limit' for concurrent mapfile
reader controlling. the eviction policy need considering these 2 limitation together. 




      was (Author: ln@webcate.net):
    i think we need archiving 2 goals for the 'open reader limit':
1. fit xceiverCount limitation of hadoop, see HBASE-24
2. control memory usage of mapfile indexes.

we may got 2 by control 'concurrent open readers', but it is not efficient. because mapfile
index size is very different. for regionserver stable, we should make sure 'open file limit'
* 'max  mapfile index size' < 'memory limit of regionserver'. this mean we have to set
the 'open file limit' low enough.

so besides 'open reader limit', there should a 'max index size limit' for concurrent mapfile
reader controlling. the eviction policy should considering these 2 limitation together. 



  
> Concurrent "open mapfile reader" limit
> --------------------------------------
>
>                 Key: HBASE-823
>                 URL: https://issues.apache.org/jira/browse/HBASE-823
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> Over in HBASE-745, Luo Ning profiling found that the number of open Readers has direct
impact on memory used.  This issue is about putting an upper bound on the number of open Readers
doing something like a bounded pool w/ a LRU eviction policy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message