hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Thrift "hang ups" with no apparent reason
Date Thu, 02 Feb 2012 19:27:23 GMT
It seems like the thrift servers are doing "something", I see they are
reading inputs from your application and one is scanning.

Earlier you mentioned that setting bigger heaps only delayed the
issue, so it seems there's a memory leak.

Which HBase version are you using? Earlier 0.90 versions had some
issues with connection handling but we've been running in production
with Thrift for ages and didn't see this issue (we migrated from the
0.89 serie to 0.90.2 so maybe you are using something older?).

Did you enable GC logging? When it "gets stuck", I'm pretty sure that
the GC log is filled with multi-seconds Full GCs.

Finally, if you are using a recent version and do see heavy GCing, try
to get a heap dump and see where the memory is allocated. "jvisualvm"
can help you doing that if you don't feel like paying for jprofiler.

Hope this helps,

J-D

On Thu, Feb 2, 2012 at 1:33 AM, Galed Friedmann
<galed.friedmann@onavo.com> wrote:
> Uploaded to pastebin:
> http://pastebin.com/YAHLyEMV
> http://pastebin.com/HuAhypsU
> http://pastebin.com/BupMyENi
>
> Thanks
>
> On Thu, Feb 2, 2012 at 10:57 AM, <yuzhihong@gmail.com> wrote:
>
>> I don't think the attachments went through.
>> Can you find some other place to upload the files ?
>>
>> Thanks
>>
>>
>>
>> On Feb 1, 2012, at 11:59 PM, Galed Friedmann <galed.friedmann@onavo.com>
>> wrote:
>>
>> > Hi again,
>> > Moved one of the services to another Thrift gateway and still got
>> timeouts from that service, something tells me even load balancing won't
>> help.
>> >
>> > I'm attaching 3 dumps from the Thrift servers, thrift2.dump is the
>> additional server we brought up, the other 2 files are from the Thrift that
>> is running on the HMaster.
>> >
>> > Thanks again for the help and patience.
>> >
>> > On Wed, Feb 1, 2012 at 7:23 PM, Stack <stack@duboce.net> wrote:
>> > On Wed, Feb 1, 2012 at 9:08 AM, Galed Friedmann
>> > <galed.friedmann@onavo.com> wrote:
>> > > Can you explain how to take the dump from the Thrift server? I couldn't
>> > > find how to do that.
>> > >
>> >
>> > Try this:
>> http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
>> >
>> > > At the moment we have only 1 Thrift gateway, I'm going to add some more
>> > > with load balancing.
>> > >
>> >
>> > At a minimum, it might put off the hang.
>> > St.Ack
>> >
>>

Mime
View raw message