hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Hbase/Troubleshooting" by JeanAdrien
Date Tue, 24 Feb 2009 10:20:45 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by JeanAdrien:
http://wiki.apache.org/hadoop/Hbase/Troubleshooting

The comment on the change is:
Added JVM GC settings advise

------------------------------------------------------------------------------
  == 7. Problem: DFS instability and/or regionserver lease timeouts ==
   * HBase regionserver leases expire during start up
   * HBase daemons cannot find block locations in HDFS during start up or other periods of
load
+  * HBase regionserver restarts after beeing unable to report to master:
+ {{{
+ 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten
times longer than scheduled: 10000
+ 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten
times longer than scheduled: 15000
+ 2009-02-24 10:01:36,472 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable
to report to master for xxx milliseconds - retrying
+ }}}
  === Causes ===
   * Excessive connection establishment latency (HRPC sets up connections on demand)
   * Slow host name resolution
   * Network bandwidth overcommitment
+  * Very long garbage collector task for the RegionServer JVM: The ''default garbage collector''
of the HotspotTM JavaTM Virtual Machine runs a ''full gc'' on the ''old space'' when the memory
space is full, which can represent about 90% of the allocated heap space. During this task,
the running program is stopped, the timers as well. If the heap space is mostly in the swap
partition, and moreover if it is larger than the physical memory, the subsequent swap can
yield to I/O overload and takes several minutes.
  === Resolution ===
   * Insure that host name resolution latency is low, or use static entries in /etc/hosts
   * Monitor the network and insure that adequate bandwidth is available for HRPC transactions
+  * In accordance with your hardware, tune your heap space / garbage collector settings in
the HBASE_OPTS variable of {{{$HBASE_CONF/hbase-env.sh}}}. Try the ''concurrent garbage collector''
{{{(-XX:+UseConcMarkSweepGC)}}} to avoid to stop the threads during GC. Read these articles
for more info about Hotspot GC settings
+     * [http://java.sun.com/docs/hotspot/gc1.4.2/faq.html Garbage collector FAQ] Quick overview
+     * [http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html Tuning garbage
collector in Java SE 6]
  

Mime
View raw message