hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Garbage collection issues
Date Fri, 18 May 2012 17:03:29 GMT
On Fri, May 18, 2012 at 4:54 AM, Simon Kelly <simongdkelly@gmail.com> wrote:
> Cluster details
> CPUs: 4 x 2.39 GHz
> RAM: 8 GB
> Ubuntu 10.04.2 LTS
>
> 1230 - hbase logs for server1 record: We slept 13318ms instead of 3000ms and
> regionserver1 is killed by master, request_processing_time goes way up
> 1326 - hbase logs for server3 record: We slept 77377ms instead of 3000ms and
> regionserver2 is killed by master

The mailing list software strips attachments, if you want to make
files available you'll have to host them somewhere and provide URLs.

You are overcommitting this cluster. Add more RAM, consider 3-4 times more.

The standard bricks we use in production at this point have 64 GB of
RAM, so we can support HBase heaps of 8 GB and all of the other Hadoop
and system daemons comfortably, as well as concurrent MapReduce tasks
and such.

Consider:

- The CMS collector basically trades heap space for GC time. Too
little space and you are still going to get stop-the-world full GC
pauses as fragmentation overwhelms collection.

- If any of the JVM pages go to swap then during GC some page access
times will go from RAM latency to disk latency, it's pathological. You
should provision sufficient RAM so all of the -Xmx of Hadoop/HBase
daemons and the memory requirements of all other processes fit
comfortably, with allowance for overage for administrative actions.

- Writes must be buffered in RAM during housekeeping actions like
flushing and compacting. High write load scenarios mean sufficient
space for such buffering is critical.

- The amount of heap alloted for memstore is fixed by configuration.
HBase maintains this global limit as part of a strategy to avoid out
of memory conditions. Therefore, as the number of regions grow, the
available space for each region's memstore shrinks proportionally. If
you have a heap sized too small for region hosting demand, then when
the number of regions gets up there, HBase will be flushing constantly
tiny files and compacting endlessly.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

Mime
View raw message