hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thakrar, Jayesh" <jthak...@conversantmedia.com>
Subject RE: Region server crashes after GC pause time
Date Fri, 22 Apr 2016 15:52:02 GMT
Sure, here you go.
Note that a number of them are from "optimal settings" available in a number of presentations.

-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC
-XX:MaxTenuringThreshold=2 -XX:SurvivorRatio=8 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768
-XX:NewSize=750m -XX:MaxNewSize=750m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UnlockExperimentalVMOptions -XX:+PrintTenuringDistribution
-Xloggc:/home/vchadoop/var/logs/hbase/regionserver-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=4M -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
-XX:+LogVMOutput -XX:LogFile=/home/vchadoop/var/logs/hbase/vm.log -XX:+PrintSafepointStatistics
 -XX:PrintSafepointStatisticsCount=1 -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=200



-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Friday, April 22, 2016 9:56 AM
To: user@hbase.apache.org
Cc: karthi keyan <karthi93.sankar@gmail.com>; dingdongchao@baidu.com
Subject: Re: Region server crashes after GC pause time

Jayesh:
Is it possible for you to share the JVM parameters ?

Thanks

On Fri, Apr 22, 2016 at 7:48 AM, Thakrar, Jayesh < jthakrar@conversantmedia.com> wrote:

> Karthik,
>
> Yes, tuning can help - but the biggest help is to give "sufficient" 
> memory to the regionserver.
> And "sufficient" is relative - e.g. we have a 75GB heap (our increases 
> were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB) Note that we had 
> started off with 45 GB RAM and are now running on 148 GB RAM servers.
>
> Now we are very stable at 75GB and with appropriate JVM tuning our GCs 
> are well contained (sub-100 ms).
>
> Also, although I like G1, it did not fare well for us at 75 GB - with 
> the best tuning we could get upto 20-25 second pauses (HBase 1.0).
> CMS seems to work best for us.
>
> Jayesh
>
> -----Original Message-----
> From: karthi keyan [mailto:karthi93.sankar@gmail.com]
> Sent: Friday, April 22, 2016 7:04 AM
> To: user@hbase.apache.org; dingdongchao@baidu.com
> Subject: Re: Region server crashes after GC pause time
>
> Hi Ding,
>
> I have increased the Heap to 2G , still getting out of memory exception .
> Actually i had write the data to HBase at 40K writes/sec .
>  Is there any parameter to tune up , as my knowledge "- 
> XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
> Is there any other parameter required to resolve this issue???
>
> Best,
> Karthik
>
> On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao 
> <dingdongchao@baidu.com>
> wrote:
>
> > Dump the jvm heap,analysis the the heap and find which query(s) cost 
> > so many memory?
> > In my ever bad case,the RS crashed for Long GC pauses because of a 
> > big query on Batch Get operation.
> >
> >
> > In addition,I think you can increase the men of JVM, 512m is so 
> > small for RS.
> >
> >
> >
> > 在 16/4/14 14:00, "karthi keyan" <karthi93.sankar@gmail.com> 写入:
> >
> > >Hi ,
> > >
> > >i got this issue in HBase while at peak time handling more requests .
> > >can any one pls guide me to resolve the Long GC pauses in hbase .
> > >
> > >JDK-7 , JVM heap 512m
> > >
> > >HBase 0.98.13
> > >
> > >
> > > INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in 
> > >JVM or host machine (eg GC): pause of approximately 1466ms GC pool 
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms  INFO 
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or 
> > >host machine (eg GC): pause of approximately 2304ms GC pool 
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms  INFO 
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or 
> > >host machine (eg GC): pause of approximately 2287ms GC pool 
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > >
> > > INFO  [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from 
> > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket 
> > >connection and attempting reconnect  INFO 
> > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from 
> > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket 
> > >connection and attempting reconnect
> > >
> > >Once after this HBase Region Server moved to Dead state.
> > >
> > >Best,
> > >Karthik
> >
> >
>
>
>
>
> This email and any files included with it may contain privileged, 
> proprietary and/or confidential information that is for the sole use 
> of the intended recipient(s).  Any disclosure, copying, distribution, 
> posting, or use of the information contained in or attached to this 
> email is prohibited unless permitted by the sender.  If you have 
> received this email in error, please immediately notify the sender via 
> return email, telephone, or fax and destroy this original transmission 
> and its included files without reading or saving it in any manner.
> Thank you.
>




This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Mime
View raw message