hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rendon, Carlos (KBB - Irvine)" <Carlos.Ren...@kbb.com>
Subject RE: Region Server OutOfMemory Error
Date Fri, 09 Jan 2015 20:24:56 GMT
We increased heap by 50%.

After re-reading your message I'm not sure it's the same issue even I ran into OOME crashes
with same message as yours. 
My crashes were proceeded by very long garbage collection times and JVM GC logs had "to space
exhausted" messages. That doesn't seem to match your description.

Did you check if you just configured for more memory than was actually available on your machine?

-Carlos

-----Original Message-----
From: Shuai Lin [mailto:linshuai2012@gmail.com] 
Sent: Thursday, January 08, 2015 5:28 PM
To: user@hbase.apache.org
Subject: Re: Region Server OutOfMemory Error

Hi Rendon,

Thanks for sharing! I'd like to know how much heap did you give to each RS (before and after
you fix the problem)? Does increasing the heap size works well for you?

Regads,
Shuai

On Fri, Jan 9, 2015 at 1:36 AM, Rendon, Carlos (KBB - Irvine) < Carlos.Rendon@kbb.com>
wrote:

> I recently ran into this exact same issue on G1GC. In my case I had 
> the luxury of giving HBase more heap space.
> If that is an option for you, you might try it out and see if it helps.
>
> -Carlos
>
> -----Original Message-----
> From: Shuai Lin [mailto:linshuai2012@gmail.com]
> Sent: Tuesday, January 06, 2015 3:33 AM
> To: user@hbase.apache.org
> Subject: Region Server OutOfMemory Error
>
> Hi all,
>
> We have a hbase cluster of 5 region servers, each, each hosting 60+ 
> regions.
>
> But under heavy load the region servers crashes for OOME now and then:
>
> #
> # java.lang.OutOfMemoryError: Java heap space # 
> -XX:OnOutOfMemoryError="kill -9 %p"
> #   Executing /bin/sh -c "kill -9 16820"...
>
> We have max heap size set to 22GB (-Xmx22528m) for each RS, and uses 
> the G1GC (-XX:+UseG1GC). To debug the problem we have turned on the 
> jvm GC log.  The last few lines of the GC log before each crash are 
> always like
> this:
>
> 2015-01-06T11:10:19.087+0000: 5035.720: [Full GC 7122M->5837M(21G),
> 0.8867660 secs]
>    [Eden: 1024.0K(7278.0M)->0.0B(8139.0M) Survivors: 68.0M->0.0B Heap:
> 7122.7M(22.0G)->5837.2M(22.0G)]
>  [Times: user=1.42 sys=0.00, real=0.89 secs]
> 2015-01-06T11:10:19.976+0000: 5036.608: [Full GC 5837M->5836M(21G),
> 0.6378260 secs]
>    [Eden: 0.0B(8139.0M)->0.0B(8139.0M) Survivors: 0.0B->0.0B Heap:
> 5837.2M(22.0G)->5836.5M(22.0G)]
>  [Times: user=0.93 sys=0.00, real=0.63 secs]
>
> From the last lineI see the heap only occupies 5837MB, and the 
> capacity is 22GB, so how can the OOM happen? Or is my interpretation 
> of the gc log wrong?
>
> I read some articles and onlhy got some basic concept of G1GC. I've 
> tried tools like GCViewer, but none gives me useful explanation of the 
> details of the GC log.
>
>
> Regards,
> Shuai
>
Mime
View raw message