hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Java Commited Virtual Memory significally larged then Heap Memory
Date Thu, 13 Jan 2011 06:42:37 GMT
On Wed, Jan 12, 2011 at 5:01 PM, Tatsuya Kawano <tatsuya6502@gmail.com>wrote:

> > And
> > in some circumstances (like all the rigged tests I've attempted to do)
> these
> > get cleaned up nicely by the JVM. It seems only in pretty large heaps in
> > real workloads does the leak actually end up running away.
>
> This issue should be circumstance dependent as we don't have direct control
> on deallocating those buffers. We need them GCed but they never occupy the
> Java heap to encourage the GC to run.
>

Thanks to reflection and use of undocumented APIs, you can actually free() a
direct buffer - check out the patch referenced earlier in this thread.

Of course it probably doesn't work on other JVMs... oh well.

-Todd

>
>
> On Jan 13, 2011, at 8:30 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > On Wed, Jan 12, 2011 at 3:25 PM, Tatsuya Kawano <tatsuya6502@gmail.com
> >wrote:
> >
> >> Hi Friso and everyone,
> >>
> >> OK. We don't have to spend time to juggle hadoop-core jars anymore since
> >> Todd is working hard on enhancing hadoop-lzo behavior.
> >>
> >> I think your assumption is correct, but what I was trying to say was
> HBase
> >> doesn't change the way to use Hadoop compressors since HBase 0.20
> release
> >> while Hadoop added reinit() on 0.21. I verified that ASF Hadoop 0.21 and
> >> CDH3b3 have reinit() and ASF Hadoop 0.20.2 (including its append branch)
> and
> >> CDH3b2 don't. I saw you had no problem running HBase 0.89 on CDH3b2, so
> I
> >> thought HBase 0.90 would work fine on ASF Hadoop 0.20.2. Because both of
> >> them don't have reinit().
> >>
> >>
> > Yep - but that jar isn't wire-compatible with a CDH3b3 cluster. So if you
> > have a CDH3b3 cluster for one of the other features included, you need to
> > use a 3b3 client jar as well, which includes the reinit stuff.
> >
> >
> >> HBase tries to create an output compression stream on each compression
> >> block, and one HFile flush will contain roughly 1000 compression blocks.
> I
> >> think reinit() could get called 1000 times on one flush, and if
> hadoop-lzo
> >> allocates 64MB block on reinit() (HBase's compression blocks is about
> 64KB
> >> though), it will become pretty much something you're observing now.
> >>
> >>
> > Yep - though I think it's only leaking a 64K buffer for each in 0.4.8.
> And
> > in some circumstances (like all the rigged tests I've attempted to do)
> these
> > get cleaned up nicely by the JVM. It seems only in pretty large heaps in
> > real workloads does the leak actually end up running away.
> >
> > -Todd
> >
> >>
> >> On Jan 13, 2011, at 7:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
> >>
> >>> Can someone who is having this issue try checking out the following git
> >>> branch and rebuilding LZO?
> >>>
> >>> https://github.com/toddlipcon/hadoop-lzo/tree/realloc
> >>>
> >>> This definitely stems one leak of a 64KB directbuffer on every reinit.
> >>>
> >>> -Todd
> >>>
> >>> On Wed, Jan 12, 2011 at 2:12 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> >>>
> >>>> Yea, you're definitely on the right track. Have you considered systems
> >>>> programming, Friso? :)
> >>>>
> >>>> Hopefully should have a candidate patch to LZO later today.
> >>>>
> >>>> -Todd
> >>>>
> >>>> On Wed, Jan 12, 2011 at 1:20 PM, Friso van Vollenhoven <
> >>>> fvanvollenhoven@xebia.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>> My guess is indeed that it has to do with using the reinit() method
> on
> >>>>> compressors and making them long lived instead of throwaway together
> >> with
> >>>>> the LZO implementation of reinit(), which magically causes NIO buffer
> >>>>> objects not to be finalized and as a result not release their native
> >>>>> allocations. It's just theory and I haven't had the time to properly
> >> verify
> >>>>> this (unfortunately, I spend most of my time writing application
> code),
> >> but
> >>>>> Todd said he will be looking into it further. I browsed the LZO
code
> to
> >> see
> >>>>> what was going on there, but with my limited knowledge of the HBase
> >> code it
> >>>>> would be bald to say that this is for sure the case. It would be
my
> >> first
> >>>>> direction of investigation. I would add some logging to the LZO
code
> >> where
> >>>>> new direct byte buffers are created to log how often that happens
and
> >> what
> >>>>> size they are and then redo the workload that shows the leak.
> Together
> >> with
> >>>>> some profiling you should be able to see how long it takes for these
> >> get
> >>>>> finalized.
> >>>>>
> >>>>> Cheers,
> >>>>> Friso
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 12 jan 2011, at 20:08, Stack wrote:
> >>>>>
> >>>>>> 2011/1/12 Friso van Vollenhoven <fvanvollenhoven@xebia.com>:
> >>>>>>> No, I haven't. But the Hadoop (mapreduce) LZO compression
is not
> the
> >>>>> problem. Compressing the map output using LZO works just fine. The
> >> problem
> >>>>> is HBase LZO compression. The region server process is the one with
> the
> >>>>> memory leak...
> >>>>>>>
> >>>>>>
> >>>>>> (Sorry for dumb question Friso) But HBase is leaking because
we make
> >>>>>> use of the Compression API in a manner that produces leaks?
> >>>>>> Thanks,
> >>>>>> St.Ack
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Todd Lipcon
> >>>> Software Engineer, Cloudera
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message