hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: RegionSevevers repeatedly getting killed with GC Pause and Zookeeper timeout
Date Tue, 13 Dec 2016 07:08:53 GMT
On Mon, Dec 12, 2016 at 10:22 PM, Sandeep Reddy <sandeepvreddy@outlook.com>

> Hi,
> From past one week multiple regionservers are getting killed due to GC
> pause and zookeeper timeouts exceptions.
And previous to this week, all ran fine?

If so, do you know what changed?

> We are trying to tune GC values with different HBASE_OPTS, but no luck.
> Following is our regionserver configuration:
>   *   Hardware: 24GB RAM with 8 Cores
>   *
> hbase-env.sh configuration:
> export HBASE_STOP_TIMEOUT="180"
> export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
> CMSInitiatingOccupancyFraction=70 -Xmn256m -XX:+
> UseCMSInitiatingOccupancyOnly"
> export HBASE_OPTS="$HBASE_OPTS -XX:MaxNewSize=128m -XX:NewSize=128m"
> export HBASE_REGIONSERVER_OPTS="-Xmx5g -Xms5g"
> export HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize=5g"
> export HBASE_MANAGES_ZK=falseexport HBASE_HEAPSIZE=4096
>   *   zookeeper tickTime=2000
There's a bunch that has been written on tuning GC and GC in HBase in
particular if you poke around on the net.

Can you give the server more heap? Above it says 5G. Can you go up from
there or are you up against some ceiling.

What does it say caused the Full GC? Is it always the same thing?

Do you get a heap given you run with  -XX:+HeapDumpOnOutOfMemoryError  ? If
so, open it in a profiler (JMC?). Usually the object retainer can be
plainly seen.


> Thanks,
> Sandeep.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message