hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Hbase/Troubleshooting" by AndrewPurtell
Date Sat, 11 Apr 2009 20:07:50 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by AndrewPurtell:
http://wiki.apache.org/hadoop/Hbase/Troubleshooting

The comment on the change is:
Added item 9: ZooKeeper SessionExpired events

------------------------------------------------------------------------------
   1. [#6 Problem: "No live nodes contain current block"]
   1. [#7 Problem: DFS instability and/or regionserver lease timeouts]
   1. [#8 Problem: Instability on Amazon EC2]
+  1. [#9 Problem: Zookeeper SessionExpired events]
  
  [[Anchor(1)]]
  == 1. Problem: Master initializes, but Region Servers do not ==
@@ -126, +127 @@

   * In accordance with your hardware, tune your heap space / garbage collector settings in
the HBASE_OPTS variable of {{{$HBASE_CONF/hbase-env.sh}}}. Try the ''concurrent garbage collector''
{{{(-XX:+UseConcMarkSweepGC)}}} to avoid to stop the threads during GC. Read these articles
for more info about Hotspot GC settings
      * [http://java.sun.com/docs/hotspot/gc1.4.2/faq.html Garbage collector FAQ] Quick overview
      * [http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html Tuning garbage
collector in Java SE 6]
-  * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
}}}
+  * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:ParallelGCThreads=8 }}}
  
  [[Anchor(8)]]
  == 8. Problem: Instability on Amazon EC2 ==
@@ -141, +142 @@

   * Use X-Large instances if you are also running mappers and reducers co-located with system
daemons.
   * Consider splitting storage and computational function over disjoint instance sets. 
  
+ [[Anchor(9)]]
+ == 9. Problem: ZooKeeper SessionExpired events ==
+  * Master or RegionServers reinitialize their ZooKeeper wrappers after receiving SessionExpired
events.
+  * Master or RegionServer ephemeral nodes disappear while the node is still otherwise functional.
+ === Causes ===
+  * Java GC is starving the ZooKeeper heartbeat thread.
+ === Resolution ===
+  * Increase the session timeout. For example, add the following to your hbase-site.xml to
increase the timeout from the default of 10 seconds to 60 seconds.
+ {{{
+   <property>
+     <name>zookeeper.session.timeout</name>
+     <value>60000</value>
+   </property>
+ }}}
+  * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:ParallelGCThreads=8 }}}
+ 

Mime
View raw message