hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop Sam John <anoo...@huawei.com>
Subject RE: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space
Date Tue, 12 Feb 2013 06:44:02 GMT
Asaf,
           You have created a wrapper around the original InternalScanner instance created
by the compaction flow?

>Where do the KV generated during the compaction process queue up before being written
to the disk? Is this buffer configurable?
When I wrote the Region Observer my assumption was the the compaction process works in Streaming
fashion, thus even if I decide to generate a KV per KV I see, it still shouldn't be a problem
memory wise.

There is no queuing. Your assumption is correct only. It is written to the writer as and when.
(Just like how memstore flush doing the HFile write) As Lars said a look at your code can
tell if some thing is going wrong.  Do you have blooms being used?

-Anoop-
________________________________________
From: Mesika, Asaf [asaf.mesika@gmail.com]
Sent: Tuesday, February 12, 2013 11:16 AM
To: user@hbase.apache.org
Subject: Custom preCompact RegionObserver crashes entire cluster on OOME: Heap Space

Hi,

I wrote a RegionObserver which does preCompact.
I activated in pre-production, and then entire cluster dropped dead: One RegionServer after
another crashed on OutOfMemoryException: Heap Space.

My preCompact method generates a KeyValue per each set of Column Qualifiers it sees.
When I remove the coprocessor and restart the cluster, cluster remains stable.
I have 8 RS, each has 4 GB Heap. There about 9 regions (from a specific table I'm working
on) per Region Server.
Running HBase 0.94.3

The crash occur when the major compaction fires up, apparently cluster wide.


My question is this: Where do the KV generated during the compaction process queue up before
being written to the disk? Is this buffer configurable?
When I wrote the Region Observer my assumption was the the compaction process works in Streaming
fashion, thus even if I decide to generate a KV per KV I see, it still shouldn't be a problem
memory wise.

Of course I'm trying to improve my code so it will generate much less new KV (by simply altering
the existing KVs received from the InternalScanner).

Thank you,

Asaf
Mime
View raw message