Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <AANLkTinipg+3ANq9dggyTiBL5zCqKy+jp9F2W0JD7t4N@mail.gmail.com>
References: <AANLkTi=p3osjGXJ+LAFJhAHBF_aFGWjCso0Fz6ry+Naq@mail.gmail.com>
 <AANLkTimMtnv2RMWz1Ug=6YK_L5Y=pCwcFVF5dbn-FbjQ@mail.gmail.com>
 <61DF20EF-BF0A-4160-B06F-A4E8B9745B6D@xebia.com>
 <AANLkTikbSmXi4u+Wk+svK9dQfgAZyGZYM+VJmQTQopPh@mail.gmail.com>
 <4D2C88AB.4000706@mozilla.com>
 <AANLkTikSvGgz_ap4_EyUX35ga61QSdz0xMOrmX05JBhg@mail.gmail.com>
 <AANLkTikJaWCqnNrU8L1v9BqSZNBwuFzUZX-St_iNrUeb@mail.gmail.com>
 <B97C4DD0-A234-4851-971C-608E320B7F3B@xebia.com>
 <AANLkTimwPtQNWrjL5v8kucyMs+KL-TJB=4jMmGomQp75@mail.gmail.com>
 <6B358FAD-B718-4904-8DB4-7BBDA50F54D9@xebia.com>
 <85972BC7-7919-4978-9F33-5A1DAC2F21EC@gmail.com>
 <2BF42DA7-AEFB-40D9-B0A8-0C555F0941DE@xebia.com>
 <AANLkTi=5r03oeRJtdA4fqseAvV3MbB5c2yp9VMN3oCLi@mail.gmail.com>
 <310F9349-EA38-405A-86B9-B4582939FD7F@xebia.com>
 <AANLkTinipg+3ANq9dggyTiBL5zCqKy+jp9F2W0JD7t4N@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 12 Jan 2011 14:50:42 -0800
Message-ID: <AANLkTimZbsNqK98B+BeU3=G4XE5aJiwKaeAptkkpbF3a@mail.gmail.com>
Subject: Re: Java Commited Virtual Memory significally larged then Heap Memory
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a4f4105c2d0499ae09eb

--90e6ba53a4f4105c2d0499ae09eb
Content-Type: text/plain; charset=ISO-8859-1

Can someone who is having this issue try checking out the following git
branch and rebuilding LZO?

https://github.com/toddlipcon/hadoop-lzo/tree/realloc

This definitely stems one leak of a 64KB directbuffer on every reinit.

-Todd

On Wed, Jan 12, 2011 at 2:12 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Yea, you're definitely on the right track. Have you considered systems
> programming, Friso? :)
>
> Hopefully should have a candidate patch to LZO later today.
>
> -Todd
>
> On Wed, Jan 12, 2011 at 1:20 PM, Friso van Vollenhoven <
> fvanvollenhoven@xebia.com> wrote:
>
>> Hi,
>> My guess is indeed that it has to do with using the reinit() method on
>> compressors and making them long lived instead of throwaway together with
>> the LZO implementation of reinit(), which magically causes NIO buffer
>> objects not to be finalized and as a result not release their native
>> allocations. It's just theory and I haven't had the time to properly verify
>> this (unfortunately, I spend most of my time writing application code), but
>> Todd said he will be looking into it further. I browsed the LZO code to see
>> what was going on there, but with my limited knowledge of the HBase code it
>> would be bald to say that this is for sure the case. It would be my first
>> direction of investigation. I would add some logging to the LZO code where
>> new direct byte buffers are created to log how often that happens and what
>> size they are and then redo the workload that shows the leak. Together with
>> some profiling you should be able to see how long it takes for these get
>> finalized.
>>
>> Cheers,
>> Friso
>>
>>
>>
>> On 12 jan 2011, at 20:08, Stack wrote:
>>
>> > 2011/1/12 Friso van Vollenhoven <fvanvollenhoven@xebia.com>:
>> >> No, I haven't. But the Hadoop (mapreduce) LZO compression is not the
>> problem. Compressing the map output using LZO works just fine. The problem
>> is HBase LZO compression. The region server process is the one with the
>> memory leak...
>> >>
>> >
>> > (Sorry for dumb question Friso) But HBase is leaking because we make
>> > use of the Compression API in a manner that produces leaks?
>> > Thanks,
>> > St.Ack
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba53a4f4105c2d0499ae09eb--