Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Friso van Vollenhoven <fvanvollenhoven@xebia.com>
To: "<user@hbase.apache.org>" <user@hbase.apache.org>
Subject: Re: Java Commited Virtual Memory significally larged then Heap
 Memory
Thread-Topic: Java Commited Virtual Memory significally larged then Heap
 Memory
Thread-Index: 
 AQHLp/fjsSancgzbfEyXgMd6ROgvlZPL3vQAgAAEloCAAAMsAIAAFeiAgABGZQCAAKU5gIAAJ86AgAAOGwCAAA6TAIAADtQAgAADwQCAAHgoAIAAJQoA
Date: Wed, 12 Jan 2011 21:20:56 +0000
Message-ID: <310F9349-EA38-405A-86B9-B4582939FD7F@xebia.com>
References: <AANLkTi=p3osjGXJ+LAFJhAHBF_aFGWjCso0Fz6ry+Naq@mail.gmail.com>
 <AANLkTimMtnv2RMWz1Ug=6YK_L5Y=pCwcFVF5dbn-FbjQ@mail.gmail.com>
 <61DF20EF-BF0A-4160-B06F-A4E8B9745B6D@xebia.com>
 <AANLkTikbSmXi4u+Wk+svK9dQfgAZyGZYM+VJmQTQopPh@mail.gmail.com>
 <4D2C88AB.4000706@mozilla.com>
 <AANLkTikSvGgz_ap4_EyUX35ga61QSdz0xMOrmX05JBhg@mail.gmail.com>
 <AANLkTikJaWCqnNrU8L1v9BqSZNBwuFzUZX-St_iNrUeb@mail.gmail.com>
 <B97C4DD0-A234-4851-971C-608E320B7F3B@xebia.com>
 <AANLkTimwPtQNWrjL5v8kucyMs+KL-TJB=4jMmGomQp75@mail.gmail.com>
 <6B358FAD-B718-4904-8DB4-7BBDA50F54D9@xebia.com>
 <85972BC7-7919-4978-9F33-5A1DAC2F21EC@gmail.com>
 <2BF42DA7-AEFB-40D9-B0A8-0C555F0941DE@xebia.com>
 <AANLkTi=5r03oeRJtdA4fqseAvV3MbB5c2yp9VMN3oCLi@mail.gmail.com>
In-Reply-To: <AANLkTi=5r03oeRJtdA4fqseAvV3MbB5c2yp9VMN3oCLi@mail.gmail.com>
Accept-Language: nl-NL, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <64c61817-5b81-44ab-95f8-d7e82ec15ca5>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,
My guess is indeed that it has to do with using the reinit() method on comp=
ressors and making them long lived instead of throwaway together with the L=
ZO implementation of reinit(), which magically causes NIO buffer objects no=
t to be finalized and as a result not release their native allocations. It'=
s just theory and I haven't had the time to properly verify this (unfortuna=
tely, I spend most of my time writing application code), but Todd said he w=
ill be looking into it further. I browsed the LZO code to see what was goin=
g on there, but with my limited knowledge of the HBase code it would be bal=
d to say that this is for sure the case. It would be my first direction of =
investigation. I would add some logging to the LZO code where new direct by=
te buffers are created to log how often that happens and what size they are=
 and then redo the workload that shows the leak. Together with some profili=
ng you should be able to see how long it takes for these get finalized.

Cheers,
Friso


On 12 jan 2011, at 20:08, Stack wrote:

> 2011/1/12 Friso van Vollenhoven <fvanvollenhoven@xebia.com>:
>> No, I haven't. But the Hadoop (mapreduce) LZO compression is not the pro=
blem. Compressing the map output using LZO works just fine. The problem is =
HBase LZO compression. The region server process is the one with the memory=
 leak...
>>=20
>=20
> (Sorry for dumb question Friso) But HBase is leaking because we make
> use of the Compression API in a manner that produces leaks?
> Thanks,
> St.Ack