Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 209.85.210.185 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Pz38NIRXwv9eVulu3/5+tJxotdW3PEN9ek45wq04CMJ4codSfWhYn13cghAZgU9VPS
         LkRwh9PxZUJYLz6TGkCGr0kqs0sitcBjMEug3BNf39w6D5CAPIfu2i2nwGHEQAD+v+ns
         wJTRV/eiqOTxsBTMJOH6y6GIVJ7cv/Dh7dqE4=
MIME-Version: 1.0
In-Reply-To: <fa03480d0907092151y1ed7274eq1ee5e85418cf6744@mail.gmail.com>
References: <fa03480d0907092058w203c0ea5uf8f2c6689d4759eb@mail.gmail.com>
	 <78568af10907092129g4657a1e3g5b9125d076a9fe77@mail.gmail.com>
	 <fa03480d0907092151y1ed7274eq1ee5e85418cf6744@mail.gmail.com>
Date: Thu, 9 Jul 2009 21:55:29 -0700
Message-ID: <78568af10907092155t24f7ecc4ucbcaba66475db3c5@mail.gmail.com>
Subject: Re: Question about HBase
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

That size is not memory-resident, so the total data size is not an
issue.  The index size is what limits you with RAM, and its about 1 MB
per region (256MB region).

-ryan

On Thu, Jul 9, 2009 at 9:51 PM, zsongbo<zsongbo@gmail.com> wrote:
> Hi Ryan,
>
> Thanks.
>
> If your regionsize is about 250MB, than 400 regions can store 100GB data =
on
> each regionserver.
> Now, if you have 100TB data, then you need 1000 regionservers.
> We are not google or yahoo who have so many nodes.
>
> Schubert
>
> On Fri, Jul 10, 2009 at 12:29 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> re: #2: in fact we don't know that... I know that I ran run 200-400
>> regions on a regionserver with a heap size of 4-5gb. =A0More even. =A0I
>> bet I could have 1000 regions open on 4gb ram. =A0Each region is ~ 1mb
>> of all the time data, so there we go.
>>
>> As for compactions, they are fairly fast, 0-30s or so depending on a
>> number of factors. =A0Practically speaking it has not been a problem for
>> me, and I've put 1200 gb into hbase so far.
>>
>> On Thu, Jul 9, 2009 at 8:58 PM, zsongbo<zsongbo@gmail.com> wrote:
>> > Hi all,
>> >
>> > 1. In this configuration property:
>> >
>> > =A0<property>
>> > =A0 =A0<name>hbase.hstore.compactionThreshold</name>
>> > =A0 =A0<value>3</value>
>> > =A0 =A0<description>
>> > =A0 =A0If more than this number of HStoreFiles in any one HStore
>> > =A0 =A0(one HStoreFile is written per flush of memcache) then a compac=
tion
>> > =A0 =A0is run to rewrite all HStoreFiles files as one. =A0Larger numbe=
rs
>> > =A0 =A0put off compaction but when it runs, it takes longer to complet=
e.
>> > =A0 =A0During a compaction, updates cannot be flushed to disk. =A0Long
>> > =A0 =A0compactions require memory sufficient to carry the logging of
>> > =A0 =A0all updates across the duration of the compaction.
>> > =A0 =A0If too large, clients timeout during compaction.
>> > =A0 =A0</description>
>> > =A0</property>
>> >
>> >
>> > That says "During a compaction, updates cannot be flushed to disk."
>> > Does it mean that, when compaction, the memcache cannot be flushed to
>> disk?
>> > I think it is not good.
>> >
>> > 2. We know that HBase cannot serve too many regions on each regionserv=
er.
>> If
>> > only 200 regions(256MB), only 50GB storage can be used.
>> > I my tested whith have 1.5GB heap and 256MB regionsize, each regionser=
ver
>> > can support 150 regions, and then OutOfMem.
>> > Can anybody explain more detail here of the reason?
>> >
>> > To use more storage, can I set larger regionsize? such as 1GB, 10GB?
>> > I have worry about the compaction time would be long with so large
>> regions.
>> >
>> > Schubert
>> >
>>
>