hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region server crashes after GC pause time
Date Fri, 22 Apr 2016 14:56:13 GMT
Jayesh:
Is it possible for you to share the JVM parameters ?

Thanks

On Fri, Apr 22, 2016 at 7:48 AM, Thakrar, Jayesh <
jthakrar@conversantmedia.com> wrote:

> Karthik,
>
> Yes, tuning can help - but the biggest help is to give "sufficient" memory
> to the regionserver.
> And "sufficient" is relative - e.g. we have a 75GB heap (our increases
> were like this - 8, 12, 16, 24, 32, 45, 60 and 75 GB)
> Note that we had started off with 45 GB RAM and are now running on 148 GB
> RAM servers.
>
> Now we are very stable at 75GB and with appropriate JVM tuning our GCs are
> well contained (sub-100 ms).
>
> Also, although I like G1, it did not fare well for us at 75 GB - with the
> best tuning we could get upto 20-25 second pauses (HBase 1.0).
> CMS seems to work best for us.
>
> Jayesh
>
> -----Original Message-----
> From: karthi keyan [mailto:karthi93.sankar@gmail.com]
> Sent: Friday, April 22, 2016 7:04 AM
> To: user@hbase.apache.org; dingdongchao@baidu.com
> Subject: Re: Region server crashes after GC pause time
>
> Hi Ding,
>
> I have increased the Heap to 2G , still getting out of memory exception .
> Actually i had write the data to HBase at 40K writes/sec .
>  Is there any parameter to tune up , as my knowledge "-
> XX:CMSInitiatingOccupancyFraction=N " i tuned like this in HBase.
> Is there any other parameter required to resolve this issue???
>
> Best,
> Karthik
>
> On Thu, Apr 14, 2016 at 12:21 PM, Ding,Dongchao <dingdongchao@baidu.com>
> wrote:
>
> > Dump the jvm heap,analysis the the heap and find which query(s) cost
> > so many memory?
> > In my ever bad case,the RS crashed for Long GC pauses because of a big
> > query on Batch Get operation.
> >
> >
> > In addition,I think you can increase the men of JVM, 512m is so small
> > for RS.
> >
> >
> >
> > 在 16/4/14 14:00, "karthi keyan" <karthi93.sankar@gmail.com> 写入:
> >
> > >Hi ,
> > >
> > >i got this issue in HBase while at peak time handling more requests .
> > >can any one pls guide me to resolve the Long GC pauses in hbase .
> > >
> > >JDK-7 , JVM heap 512m
> > >
> > >HBase 0.98.13
> > >
> > >
> > > INFO  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM
> > >or host machine (eg GC): pause of approximately 1466ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=1967ms  INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> > >machine (eg GC): pause of approximately 2304ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms  INFO
> > >[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host
> > >machine (eg GC): pause of approximately 2287ms GC pool
> > >'ConcurrentMarkSweep' had collection(s): count=1 time=2775ms
> > >
> > > INFO  [RS:0;0:0:0:0:0:0:0:0:44037-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280004, closing socket
> > >connection and attempting reconnect  INFO
> > >[SplitLogWorker-<host>,44037,1460468489645-SendThread(<host>:2181)]
> > >zookeeper.ClientCnxn: Client session timed out, have not heard from
> > >server in 6819ms for sessionid 0x1540ab48b280005, closing socket
> > >connection and attempting reconnect
> > >
> > >Once after this HBase Region Server moved to Dead state.
> > >
> > >Best,
> > >Karthik
> >
> >
>
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s).  Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender.  If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message