hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Hung <YTHu...@winbond.com>
Subject suggestion for how eliminate memory problem in heavy-write hbase region server
Date Fri, 25 Apr 2014 06:07:13 GMT
Dear All,

My current hbase environment is heavy write cluster with constant 2000+ insert rows / second
spread to 10 region servers.
Each day I also need to do data deletion, and that will add a lot of IO to the cluster.

The problem is sometimes after a week, one of the region server will crash because
2014-04-10T10:17:47.200+0800: 1281486.956: [GC 1281486.956: [ParNew (promotion failed): 235959K->235959K(235968K),
0.0836790 secs]1281487.040: [CMS2014-04-10T10:21:14.957+0800: 1281694.712: [CMS-concurrent-sweep:
267.111/279.155 secs] [Times: user=334.79 sys=14.38, real=279.11 secs]
(concurrent mode failure): 13961950K->6802914K(16515072K), 209.9436660 secs] 14186496K->6802914K(16751040K),
[CMS Perm : 42864K->42859K(71816K)], 210.0274680 secs] [Times: user=210.18 sys=0.01, real=209.99
secs]

I look into the gc log and usually find some information about CMS concurrent sweep that took
a very long time to complete, such as:
2014-04-10T10:15:56.929+0800: 1281376.684: [CMS-concurrent-sweep: 48.834/58.027 secs] [Times:
user=101.52 sys=11.82, real=58.02 secs]

I do a lot of google-ing and already read the Todd Lipcon avoiding full GC, or other blogs
that sometimes tells me how to set jvm flags such as this:
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=70
-Xmn256m
-Xmx16384m
-XX:+DisableExplicitGC
-XX:+UseCompressedOops
-XX:PermSize=160m
-XX:MaxPermSize=160m
-XX:GCTimeRatio=19
-XX:SoftRefLRUPolicyMSPerMB=0
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=1
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled
-XX:CMSMaxAbortablePrecleanTime=300
-XX:+CMSScavengeBeforeRemark

But alas, the problem still exist.

I also know that java 1.7 has a new G1GC that probably can be used to fix this problem, but
I don't know if hbase 0.96 is ready to use it?

I would really appreciate it if someone out there can share one or two things about jvm configuration
to achieve a more stable region server.

Best regards,
Henry

________________________________
The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message