hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Question about HBase
Date Fri, 10 Jul 2009 04:55:29 GMT
That size is not memory-resident, so the total data size is not an
issue.  The index size is what limits you with RAM, and its about 1 MB
per region (256MB region).

-ryan

On Thu, Jul 9, 2009 at 9:51 PM, zsongbo<zsongbo@gmail.com> wrote:
> Hi Ryan,
>
> Thanks.
>
> If your regionsize is about 250MB, than 400 regions can store 100GB data on
> each regionserver.
> Now, if you have 100TB data, then you need 1000 regionservers.
> We are not google or yahoo who have so many nodes.
>
> Schubert
>
> On Fri, Jul 10, 2009 at 12:29 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> re: #2: in fact we don't know that... I know that I ran run 200-400
>> regions on a regionserver with a heap size of 4-5gb.  More even.  I
>> bet I could have 1000 regions open on 4gb ram.  Each region is ~ 1mb
>> of all the time data, so there we go.
>>
>> As for compactions, they are fairly fast, 0-30s or so depending on a
>> number of factors.  Practically speaking it has not been a problem for
>> me, and I've put 1200 gb into hbase so far.
>>
>> On Thu, Jul 9, 2009 at 8:58 PM, zsongbo<zsongbo@gmail.com> wrote:
>> > Hi all,
>> >
>> > 1. In this configuration property:
>> >
>> >  <property>
>> >    <name>hbase.hstore.compactionThreshold</name>
>> >    <value>3</value>
>> >    <description>
>> >    If more than this number of HStoreFiles in any one HStore
>> >    (one HStoreFile is written per flush of memcache) then a compaction
>> >    is run to rewrite all HStoreFiles files as one.  Larger numbers
>> >    put off compaction but when it runs, it takes longer to complete.
>> >    During a compaction, updates cannot be flushed to disk.  Long
>> >    compactions require memory sufficient to carry the logging of
>> >    all updates across the duration of the compaction.
>> >    If too large, clients timeout during compaction.
>> >    </description>
>> >  </property>
>> >
>> >
>> > That says "During a compaction, updates cannot be flushed to disk."
>> > Does it mean that, when compaction, the memcache cannot be flushed to
>> disk?
>> > I think it is not good.
>> >
>> > 2. We know that HBase cannot serve too many regions on each regionserver.
>> If
>> > only 200 regions(256MB), only 50GB storage can be used.
>> > I my tested whith have 1.5GB heap and 256MB regionsize, each regionserver
>> > can support 150 regions, and then OutOfMem.
>> > Can anybody explain more detail here of the reason?
>> >
>> > To use more storage, can I set larger regionsize? such as 1GB, 10GB?
>> > I have worry about the compaction time would be long with so large
>> regions.
>> >
>> > Schubert
>> >
>>
>

Mime
View raw message