hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Carroll <phobos...@gmail.com>
Subject Re: Struggling with Region Servers Running out of Memory
Date Thu, 01 Nov 2012 17:07:48 GMT
Java 6 update 34 can rotate GC Logs. -XX:+UseGCLogFileRotation


As for profiling memory dumps, jprofiler7, yourrkit, etc.. YMMV.

On Thu, Nov 1, 2012 at 10:01 AM, Jeff Whiting <jeffw@qualtrics.com> wrote:

> We don't have GC logging enabled (we did but gc.log would begin filling up
> the hdd and there was no way to clear it out without restarting the region
> server).  Anyway to en gc.log and keep it to a reasonable size?
> I have two separate jmap dumps of the a region server before it dies.  I
> haven't really looked into those yet.  I'll try to do that today.  I've
> typically used eclipse memory analyzer tool or netbeans.  Is there a
> profiler you'd recommend?
> ~Jeff
> On 10/30/2012 11:39 PM, Stack wrote:
>> On Mon, Oct 29, 2012 at 3:55 PM, Jeff Whiting <jeffw@qualtrics.com>
>> wrote:
>>> However what we are seeing is that our memory usage goes up slowly until
>>> the
>>> region server starts sputtering due to gc collection issues and it will
>>> eventually get timed out by zookeeper and be killed.
>>>  Hey Jeff.  You have GC logging enabled?  Might not tell you more than
>> you already know, that something is retaining more and more objects
>> over time.   You have a dumped heap?  What have you used to poke at
>> it?  You generally want to find the objects that have the deepest size
>> (Not all profilers let you do this though).  This is usually enough to
>> give you a clue.
>> Anything particular about the character of your load?  Ram asks if any
>> big cells in the mix?
>> St.Ack
>>  At this point I feel somewhat lost as to how to debug the problem. I'm
>>> not
>>> sure what to do next to figure out what is going on.  Any suggestions as
>>> to
>>> what to look for or debug where the memory is being used? I can generate
>>> heap dumps via jmap (although it effectively kills the region server)
>>> but I
>>> don't really know what to look for to see where the memory is going. I
>>> also
>>> have jmx setup on each region server and can connect to it that way.
>>> Thanks,
>>> ~Jeff
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> jeffw@qualtrics.com
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message