hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mesika, Asaf" <asaf.mes...@gmail.com>
Subject Re: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
Date Tue, 12 Feb 2013 09:30:38 GMT
Yes.
I have create my own internal scanner using the internalscanner given to me in preCompact
On Feb 12, 2013, at 8:44 AM, Anoop Sam John wrote:

> Asaf,
>           You have created a wrapper around the original InternalScanner instance created
by the compaction flow?
> 
>> Where do the KV generated during the compaction process queue up before being written
to the disk? Is this buffer configurable?
> When I wrote the Region Observer my assumption was the the compaction process works in
Streaming fashion, thus even if I decide to generate a KV per KV I see, it still shouldn't
be a problem memory wise.
> 
> There is no queuing. Your assumption is correct only. It is written to the writer as
and when. (Just like how memstore flush doing the HFile write) As Lars said a look at your
code can tell if some thing is going wrong.  Do you have blooms being used?
> 
> -Anoop-
> ________________________________________
> From: Mesika, Asaf [asaf.mesika@gmail.com]
> Sent: Tuesday, February 12, 2013 11:16 AM
> To: user@hbase.apache.org
> Subject: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
> 
> Hi,
> 
> I wrote a RegionObserver which does preCompact.
> I activated in pre-production, and then entire cluster dropped dead: One RegionServer
after another crashed on OutOfMemoryException: Heap Space.
> 
> My preCompact method generates a KeyValue per each set of Column Qualifiers it sees.
> When I remove the coprocessor and restart the cluster, cluster remains stable.
> I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific table I'm working
on) per Region Server.
> Running HBase 0.94.3
> 
> The crash occur when the major compaction fires up, apparently cluster wide.
> 
> 
> My question is this: Where do the KV generated during the compaction process queue up
before being written to the disk? Is this buffer configurable?
> When I wrote the Region Observer my assumption was the the compaction process works in
Streaming fashion, thus even if I decide to generate a KV per KV I see, it still shouldn't
be a problem memory wise.
> 
> Of course I'm trying to improve my code so it will generate much less new KV (by simply
altering the existing KVs received from the InternalScanner).
> 
> Thank you,
> 
> Asaf


Mime
View raw message