Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <28fc6650.1b6ce.14d708078d9.Coremail.c77_cn@163.com>
References: <397b8c9c.62a5.14d4b2663e1.Coremail.c77_cn@163.com>
	<CAMhkWdDD1CTNX+6XtPtK8aBR8-Pa90rzmH=iFEJK=_j6kB5jfQ@mail.gmail.com>
	<299d92ec.d243.14d64fe4ff2.Coremail.c77_cn@163.com>
	<CA+RK=_AnC7GDphLt4kkZ-p7oAcOhMtNp-KK5HMzqNHbFUH1s4A@mail.gmail.com>
	<CAGHyZ6JztmKmxUys50N3hjcMts04dreyLH=nAOvXXnOQCCpGGw@mail.gmail.com>
	<3fe21a95.131e1.14d6ef32271.Coremail.c77_cn@163.com>
	<CALte62whtEMtGxFx4VxnLfhXVNgfBWcZ5xTB6KE_jBghTVnycw@mail.gmail.com>
	<28fc6650.1b6ce.14d708078d9.Coremail.c77_cn@163.com>
Date: Wed, 20 May 2015 08:12:58 -0700
Message-ID: 
 <CADcMMgE8M8wp9t_FMw8XT1RyTmSH_5DS8BPuqOimATmVKe09ZQ@mail.gmail.com>
Subject: Re: How to know the root reason to cause RegionServer OOM?
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c12432c3c232051684dce1

--001a11c12432c3c232051684dce1
Content-Type: text/plain; charset=UTF-8

On Wed, May 20, 2015 at 1:46 AM, David chen <c77_cn@163.com> wrote:

> Thanks Ted,
> For scenario #1, can not see any clues in regionserver log file that
> denotes "kill -9" command was executed. Meanwhile, i think when JVM
> inspects regionserver process OOME, it will create a new thread to execute
> "kill -9 %p", the new thread should not write regionserver log, so the
> fact, there is not any clues in regionserver log, is normal. Right?
> For scenario #2, dmesg also did not provide any clues. But some clues were
> seen in /var/log/messages:
> ......
> May 14 12:00:38 localhost kernel: Out of memory: Kill process 22827 (java)
> score 497 or sacrifice child
> May 14 12:00:38 localhost kernel: Killed process 22827, UID 483, (java)
> total-vm:17569220kB, anon-rss:16296276kB, file-rss:240kB
> ......
> The 22827 above is regionserver PID.
> It looks like regionserver itself OOM(total-vm:17569220kB,
> anon-rss:16296276kB, the max-heap-size set is 15G), so was killed. Right?
>

Yes.


> But hbase has no heavy load in the cluster,


Doesn't matter. You allocated it a heap of 15G. The OS is looking for
memory and is at a extreme (swapping totally disabled?) so it starts
killing random processes. This is not an hbase issue. It is an
oversubscription problem. Google how to address.


> so i don't think it was killed because of itself OOME, instead i think
> because of lack of memory for other applications, so OS kill regionserver
> to run more applications.
> I currently has no evidence to prove my idea, so hope more helps. Thanks.
>

You quote all necessary evidence above.

St.Ack


>
>
>
>
>
>
>
> At 2015-05-20 10:04:19, "Ted Yu" <yuzhihong@gmail.com> wrote:
> >For scenario #1, you would see in the regionserver.out file that "kill -9
> "
> >command was applied due to OOME.
> >
> >For scenario #2, can you see if dmesg provides some clue ?
> >
> >Cheers
> >
> >On Tue, May 19, 2015 at 6:32 PM, David chen <c77_cn@163.com> wrote:
> >
> >> Thanks for guys reply, its indeed helped me.
> >> Another question, I think there are two possibilities to kill
> RegionServer
> >> process:
> >> 1. When JVM inspects that the memory, RegionServer has occupied, exceed
> >> the max-heap-size,  then JVM calls positively the command configured by
> >> option "-XX:OnOutOfMemoryError=kill -9 %p" to kill RegionServer
> process.
> >> 2. RegionServer process does not reach the max-heap-size, but new
> >> application need to allocation memory,  if lack of memory, OS will
> choose
> >> to kill some processes, RegionServer unfortunately becomes the first
> >> choice, so it  is killed by OS.
> >> Is my understanding right? If so, how to know which possibility my scene
> >> is?
> >> Any ideas can be appreciated!
> >>
>

--001a11c12432c3c232051684dce1--