hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Duxbury (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-70) [hbase] memory management
Date Tue, 05 Feb 2008 23:53:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565965#action_12565965
] 

Bryan Duxbury commented on HBASE-70:
------------------------------------

Agree, -1 on moving memcache to Region level. I like the way that memcache and storefiles
have analogous position and structure, making it easy to code against them. (One might argue
that their similar structure would allow us to generalize the manipulation methods further
so that we only need a single version of each manipulation.) 

What's difficult about managing the memory of the memcaches a level down anyways? Some particular
accounting issue?

> [hbase] memory management
> -------------------------
>
>                 Key: HBASE-70
>                 URL: https://issues.apache.org/jira/browse/HBASE-70
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: stack
>
> Each Store has a Memcache of edits that is flushed on a fixed period (It used to be flushed
when it grew beyond a limit). A Region can be made up of N Stores.  A regionserver has no
upper bound on the number of regions that can be deployed to it currently.  Add to this that
per mapfile, we have read the index into memory.  We're also talking about adding caching
of blocks and cells.
> We need a means of keeping an account of memory usage adjusting cache sizes and flush
rates (or sizes) dynamically -- using References where possible -- to accomodate deployment
of added regions.  If memory is strained, we should reject regions proffered by the master
with a resouce-constrained, or some such, message.
> The manual sizing we currently do ain't going to cut it for clusters of any decent size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message