hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Potential memory leak in client RPC timeout mechanism
Date Thu, 20 Oct 2011 20:58:58 GMT
Yeah it does that if the heap is big... mmm dev could be useful but it's a
guess, as what's eating all that mem might be too small there and buried
under the rest.

J-D

On Wed, Oct 19, 2011 at 7:00 PM, Shrijeet Paliwal
<shrijeet@rocketfuel.com>wrote:

> Stack, I have created https://issues.apache.org/jira/browse/HBASE-4633
> J-D, I had tried in past to get the dump in production (only
> environment where this is reproducible) but failed. The application
> freezes if any profiling activity is attempted.
> I can try and get you a dump from dev environment. Thanks for writing.
>
> -Shrijeet
>
> On Wed, Oct 19, 2011 at 4:23 PM, Stack <stack@duboce.net> wrote:
> > And file an issue please Shrijeet so we don't forget about it.
> > Thanks boss,
> > St.Ack
> >
> > On Fri, Oct 14, 2011 at 12:12 PM, Shrijeet Paliwal
> > <shrijeet@rocketfuel.com> wrote:
> >> Hi All,
> >>
> >> HBase version: 0.90.3 + Patches
> >> Hadoop version: CDH3u0
> >> Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
> >> https://issues.apache.org/jira/browse/HBASE-4003
> >>
> >> We have been using the 'hbase.client.operation.timeout' knob
> >> introduced in 2937 for quite some time now. It helps us enforce SLA.
> >> We have two HBase clusters and two HBase client clusters. One of them
> >> is much busier than the other.
> >>
> >> We have seen a deterministic behavior of clients running in busy
> >> cluster. Their (client's) memory footprint increases consistently
> >> after they have been up for roughly 24 hours.
> >> This memory footprint almost doubles from its usual value (usual case
> >> == RPC timeout disabled). After much investigation nothing concrete
> >> came out and we had to put a hack
> >> which keep heap size in control even when RPC timeout is enabled. Also
> >> please note , the same behavior is not observed in 'not so busy
> >> cluster.
> >>
> >> The patch is here : https://gist.github.com/1288023
> >>
> >> Can some one, who is also running RPC timeout in production under fair
> >> load, please share the experience.
> >>
> >> -Shrijeet
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message