Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.160.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=twHc15Sk+k5Ydq8tZYBox5tn4d2MWUto7PJ6cP8d7PmTrOj7EOv9UaVE83VLDTvXU9
         tcyANff+JkkSntRLckAsLUAActI1pxXy/1xbeDND9dfqb8HK77Z264d+lEgPdRLlPxvj
         +vs5wSOT8r5MzsQc+uH9yYpp3Gm0CTgLCzJsg=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <32CA4FA8-ABC8-493A-B9A3-3D85910AB756@email.com>
References: 
 <2D6136772A13B84E95DF6DA79E85A9F001364A05E05F@NSPEXMBX-A.the-lab.llnl.gov>
	<AANLkTinEeiLspNoKioU7nG41grv6VzkWdL8cM+w3LFdD@mail.gmail.com>
	<2D6136772A13B84E95DF6DA79E85A9F001364A05E078@NSPEXMBX-A.the-lab.llnl.gov>
	<AANLkTim=1VVyVS7z1rwAbEy66+sMRDpP9YM2-Hktccu4@mail.gmail.com>
	<22DD2D90-391B-4836-A9AF-993494D9E878@email.com>
	<AANLkTimF564hKadau8tLaoJRzVaCD=FVU6DtwdajyiJp@mail.gmail.com>
	<AANLkTimH34RQeScfO6H8b3xh7Sk+ABsNaG-uMitexv1f@mail.gmail.com>
	<C791E437-2E29-4077-89E9-C372A5AD8F38@email.com>
	<AANLkTikYJYPun3dvQWBAAsTas0vx+=Y6HqmkTH1yq1XG@mail.gmail.com>
	<32CA4FA8-ABC8-493A-B9A3-3D85910AB756@email.com>
Date: Wed, 9 Mar 2011 14:13:47 -0800
Message-ID: <AANLkTi=HG16yeh=AF6YVOkH0aVSrJPCVq+a=KDr6rfWg@mail.gmail.com>
Subject: Re: Blob storage
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yeah there's definitely something better we could do there, see "Too
easy to OOME a RS" https://issues.apache.org/jira/browse/HBASE-2506

J-D

On Wed, Mar 9, 2011 at 11:09 AM, Chris Tarnas <cft@email.com> wrote:
> When I get a chance to catch my breath I'll see about writing up somethin=
g on our experiences. One thing I will say - don't skimp on the nodes, you =
do not want to run out of RAM when using the large values. When running my =
dev environment in pseudo distributed mode on a laptop the system can have =
trouble (nothing that is not recoverable though) when the regionserver runs=
 out of memory dealing with the large value.
>
> -chris
>
>
>
>
> On Mar 8, 2011, at 1:54 PM, Jean-Daniel Cryans wrote:
>
>> That's pretty good stuff Chris! You know, you could be my new BFF if
>> you wrote a blog post about your current HBase setup, experiences, etc
>> :)
>>
>> J-D
>>
>> On Tue, Mar 8, 2011 at 11:25 AM, Chris Tarnas <cft@email.com> wrote:
>>> Yes, HBASE-3483 fixed the majority of our pauses, but not all - as JD p=
oints out we do experience issues related to inserting into several column =
families. Luckily inserts that have the really imbalanced column family siz=
es (mb vs kb) are few and far between, relatively speaking. We are also "th=
rottled" by going through thrift, but even then I can push our 10 node clus=
ter to over 200k requests a second.
>>>
>>> -chris
>>>
>>> On Mar 8, 2011, at 11:16 AM, Ryan Rawson wrote:
>>>
>>>> Probably the soft limit flushes, eh?
>>>> On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jdcryans@apache.org> wr=
ote:
>>>>> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cft@email.com> wrote:
>>>>>> Just as a point of reference, in one of our systems we have 500+mill=
ion
>>>> rows that have a cell in its own column family that is about usually a=
bout
>>>> 100bytes, but in about 10,000 of rows the cell can get to 300mb (avera=
ge is
>>>> probably about 30mb for the larger data). The jumbo sized data gets lo=
aded
>>>> in separately from the smaller data, although it all goes through the =
same
>>>> pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of=
 1GB
>>>> and with a max value size of 500mb. So far we have had no problems wit=
h the
>>>> larger values.
>>>>>>
>>>>>> Our largest problem was performance related to inserting into severa=
l
>>>> column families for the small sized value loads and pauses when flushi=
ng the
>>>> memstores. 0.90.1 helped quite a bit with that.
>>>>>
>>>>> Flushing is done without blocking, were the pauses you were seeing
>>>>> related to the "too many stores" issue or about the global memstore
>>>>> size?
>>>>>
>>>>> In general inserting into many families is a bad idea unless the size=
s
>>>>> are the same. The worst case is inserting a few kbs in one and a few
>>>>> mbs in the other. The reason being:
>>>>> https://issues.apache.org/jira/browse/HBASE-3149
>>>>>
>>>>> J-D
>>>
>>>
>
>