hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ZooKeeper/HBaseUseCases" by PatrickHunt
Date Thu, 05 Nov 2009 20:15:39 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "ZooKeeper/HBaseUseCases" page has been changed by PatrickHunt.
http://wiki.apache.org/hadoop/ZooKeeper/HBaseUseCases?action=diff&rev1=8&rev2=9

--------------------------------------------------

  Here's an idea, see if I got the idea right, obv would have to flesh this out more but this
is the general idea. I've chosen random paths below, obv you'd want some sort of prefix, better
names, etc...
  
  1) group membership:
-  # have a /regionservers znode
+  1. have a /regionservers znode
-  # master watches /regionservers for any child changes
+  1. master watches /regionservers for any child changes
-  # as each region server becomes available to do work (or track state if up but not avail)
it creates an ephemeral node
+  1. as each region server becomes available to do work (or track state if up but not avail)
it creates an ephemeral node
    * /regionserver/<host:port> = <status>
-  # master watches /regionserver/<host:port> and cleans up if RS goes away or changes
status
+  1. master watches /regionserver/<host:port> and cleans up if RS goes away or changes
status
  
  2) task assignment (ie dynamic configuration)
-  # have a /tables znode
+  1. have a /tables znode
-  # /tables/<regionserver by host:port> which gets created when master notices new
region server
+  1. /tables/<regionserver by host:port> which gets created when master notices new
region server
    * RS host:port watches this node for any child changes
-  # /tables/<regionserver by host:port>/<regionX> znode for each region assigned
to RS host:port
+  1. /tables/<regionserver by host:port>/<regionX> znode for each region assigned
to RS host:port
    * RS host:port watches this node in case reassigned by master, or region changes state
-  # /tables/<regionserver by host:port>/<regionX>/<state>-<seq#>
znode created by master
+  1. /tables/<regionserver by host:port>/<regionX>/<state>-<seq#>
znode created by master
    * seq ensures order seen by RS
    * RS deletes old state znodes as it transitions out, oldest entry is the current state,
always 1 or more znode here -- the current state
  
@@ -73, +73 @@

  See [[http://bit.ly/4ekN8G|this perf doc]] for some ideas, 20 clients doing 50k watches
each - 1 million watches on a single core standalone server and still << 5ms avg response
time (async ops, keep that in mind re implementation time) YMMV of course but your numbers
are well below this. 
  
  Worst-case scenarios -- say a cascade failure where all RS become disconnected and sessions
expire
-  # master will get notified of 1000 RS gone
+  1. master will get notified of 1000 RS gone
-  # master will delete all nodes in 2) - 1000 RS znodes, 100 regions each RS znode, 1 state
(typ) each of the 100 reg
+  1. master will delete all nodes in 2) - 1000 RS znodes, 100 regions each RS znode, 1 state
(typ) each of the 100 reg
    * 200k nodes deleted (hint: use async) for each RS
  
  Another worst case:
-  # some set of RS are flapping (this may actually be much worse than just dying)
+  1. some set of RS are flapping (this may actually be much worse than just dying)
    * consider running some sort of health check on RS before assigning work, in case it just
dies 
    * or, slowly ramp up the number of regions assigned to the RS, allow it to prove itself
vs dumping a number of regions on it and then have it flap... (donno enough about hbase to
comment resonably, but thing about something like this)
-  # for each RS master is deleting 200 znodes
+  1. for each RS master is deleting 200 znodes
  
  [PDH end]
  

Mime
View raw message