Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of vrodionov@carrieriq.com
 designates 204.235.122.22 as permitted sender)
From: Vladimir Rodionov <vrodionov@carrieriq.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Fri, 25 Apr 2014 10:47:56 -0700
Subject: RE: suggestion for how eliminate memory problem in heavy-write
 hbase region server
Thread-Topic: suggestion for how eliminate memory problem in heavy-write
 hbase region server
Thread-Index: Ac9gpi8/GCPN9Yv2RNSCuEWI8U46PQAB4Emc
Message-ID: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15187CA57C52@exchange2007.carrieriq.com>
References: 
 <53DC189E5FAEFA43BFA1BC02431031DB74086592E8@WECTMLBOX.winbond.com.tw>
	<CAHxLZBXcbzUxQ-M+=bg2RSaQG0QR-riZU-T85k=VmKvNPYZ7LQ@mail.gmail.com>,<CALte62zCS+UXj9hvHoNxW9vF=a4_p9p_GgCZAA93zj6Ue6oiLA@mail.gmail.com>
In-Reply-To: 
 <CALte62zCS+UXj9hvHoNxW9vF=a4_p9p_GgCZAA93zj6Ue6oiLA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I am just wondering why do you need large heaps on write - heavy cluster?
How many regions per RS do you have? Usually large heaps need for read inte=
nsive applications
to keep blocks in a cache, but now we have option to keep these blocks off =
heap (at least since 0.96)
With MemSLAB for MemStore enabled you need 2MB (by default) per region, thi=
s is what you
need to take into account first. Even 1000 regions won't eat more than 2GB =
of heap.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Friday, April 25, 2014 9:47 AM
To: user@hbase.apache.org
Subject: Re: suggestion for how eliminate memory problem in heavy-write hba=
se region server

Henry:
Please also take a look at the following thread:

http://search-hadoop.com/m/51M4jeDMyy1/GC+recommendations+for+large+Region+=
Server+heaps&subj=3DRE+GC+recommendations+for+large+Region+Server+heaps


On Thu, Apr 24, 2014 at 11:17 PM, Mikhail Antonov <olorinbant@gmail.com>wro=
te:

> Henry,
>
> http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html - tha=
t
> may give some insights.
>
> -Mikhail
>
>
> 2014-04-24 23:07 GMT-07:00 Henry Hung <YTHung1@winbond.com>:
>
> > Dear All,
> >
> > My current hbase environment is heavy write cluster with constant 2000+
> > insert rows / second spread to 10 region servers.
> > Each day I also need to do data deletion, and that will add a lot of IO
> to
> > the cluster.
> >
> > The problem is sometimes after a week, one of the region server will
> crash
> > because
> > 2014-04-10T10:17:47.200+0800: 1281486.956: [GC 1281486.956: [ParNew
> > (promotion failed): 235959K->235959K(235968K), 0.0836790
> secs]1281487.040:
> > [CMS2014-04-10T10:21:14.957+0800: 1281694.712: [CMS-concurrent-sweep:
> > 267.111/279.155 secs] [Times: user=3D334.79 sys=3D14.38, real=3D279.11 =
secs]
> > (concurrent mode failure): 13961950K->6802914K(16515072K), 209.9436660
> > secs] 14186496K->6802914K(16751040K), [CMS Perm :
> 42864K->42859K(71816K)],
> > 210.0274680 secs] [Times: user=3D210.18 sys=3D0.01, real=3D209.99 secs]
> >
> > I look into the gc log and usually find some information about CMS
> > concurrent sweep that took a very long time to complete, such as:
> > 2014-04-10T10:15:56.929+0800: 1281376.684: [CMS-concurrent-sweep:
> > 48.834/58.027 secs] [Times: user=3D101.52 sys=3D11.82, real=3D58.02 sec=
s]
> >
> > I do a lot of google-ing and already read the Todd Lipcon avoiding full
> > GC, or other blogs that sometimes tells me how to set jvm flags such as
> > this:
> > -XX:+UseParNewGC
> > -XX:CMSInitiatingOccupancyFraction=3D70
> > -Xmn256m
> > -Xmx16384m
> > -XX:+DisableExplicitGC
> > -XX:+UseCompressedOops
> > -XX:PermSize=3D160m
> > -XX:MaxPermSize=3D160m
> > -XX:GCTimeRatio=3D19
> > -XX:SoftRefLRUPolicyMSPerMB=3D0
> > -XX:SurvivorRatio=3D2
> > -XX:MaxTenuringThreshold=3D1
> > -XX:+UseFastAccessorMethods
> > -XX:+UseParNewGC
> > -XX:+UseConcMarkSweepGC
> > -XX:+CMSParallelRemarkEnabled
> > -XX:+UseCMSCompactAtFullCollection
> > -XX:CMSFullGCsBeforeCompaction=3D0
> > -XX:+CMSClassUnloadingEnabled
> > -XX:CMSMaxAbortablePrecleanTime=3D300
> > -XX:+CMSScavengeBeforeRemark
> >
> > But alas, the problem still exist.
> >
> > I also know that java 1.7 has a new G1GC that probably can be used to f=
ix
> > this problem, but I don't know if hbase 0.96 is ready to use it?
> >
> > I would really appreciate it if someone out there can share one or two
> > things about jvm configuration to achieve a more stable region server.
> >
> > Best regards,
> > Henry
> >
> > ________________________________
> > The privileged confidential information contained in this email is
> > intended for use only by the addressees as indicated by the original
> sender
> > of this email. If you are not the addressee indicated in this email or
> are
> > not responsible for delivery of the email to such a person, please kind=
ly
> > reply to the sender indicating this fact and delete all copies of it fr=
om
> > your computer and network server immediately. Your cooperation is highl=
y
> > appreciated. It is advised that any unauthorized use of confidential
> > information of Winbond is strictly prohibited; and any information in
> this
> > email irrelevant to the official business of Winbond shall be deemed as
> > neither given nor endorsed by Winbond.
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>

Confidentiality Notice:  The information contained in this message, includi=
ng any attachments hereto, may be confidential and is intended to be read o=
nly by the individual or entity to whom this message is addressed. If the r=
eader of this message is not the intended recipient or an agent or designee=
 of the intended recipient, please note that any review, use, disclosure or=
 distribution of this message or its attachments, in any form, is strictly =
prohibited.  If you have received this message in error, please immediately=
 notify the sender and/or Notifications@carrieriq.com and delete or destroy=
 any copy of this message and its attachments.