hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Hbase/MasterRewrite" by stack
Date Tue, 20 Oct 2009 23:18:17 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "Hbase/MasterRewrite" page has been changed by stack.
http://wiki.apache.org/hadoop/Hbase/MasterRewrite?action=diff&rev1=9&rev2=10

--------------------------------------------------

   * [[#scope|Design Scope]]
   * [[#design|Design]]
    * [[#all|Move all region state transitions to zookeeper]]
+   * [[#clean|Region State changes are clean, minimal, and comprehensive]]
+   * [[#state|Cluster/Table State Changes via ZK]]
+   * [[#schema|Schema]]
    * [[#distinct|In Zookeeper, a State and a Schema section]]
-   * [[#clean|State changes are clean, minimal, and comprehensive]]
-   * [[#schema|Schema]]
    * [[#balancer|Load Assignment/Balancer]]
    * [[#root|Remove -ROOT-]]
    * [[#root|Remove Heartbeat]]
@@ -35, +36 @@

  
  <<Anchor(problems)>>
  == Problems with current Master ==
- There is a good list in the [[https://issues.apache.org/jira/secure/ManageLinks.jspa?id=12434794|Issue
Links]] section of HBASE-1816.
+ There is a list in the [[https://issues.apache.org/jira/secure/ManageLinks.jspa?id=12434794|Issue
Links]] section of HBASE-1816.
  
  <<Anchor(scope)>>
  == Design Scope ==
   1. Rewrite of Master is for HBase 0.21
   1. Design for:
-    1. Regionserver loading (TODO: These numbers don't make sense -- jgray do you remember
what they were about?)
+    1. A cluster of 200 regionservers... (TODO: These numbers don't make sense -- jgray do
you remember what they were about?  See misc section below)
-      1. 200 regionservers
-      1. 32 outstanding wal logs per regionserver
-      1. 200 regions per regionserver being written to
-      1. 2GB or 30 hour full log roll
-      1. 10MB/sec write speed
-      1. 1.2M edits per 2G
-      1. 7k writes/second across cluster (?) -- whats this?  Wrong.
-      1. 1.2M edits per 30 hours?
-      1. 100 writes/sec across cluster (?) -- Whats this?  Wrong?
  	  
  <<Anchor(design)>>
  == Design ==
  
  <<Anchor(all)>>
  === Move all region state transitions to zookeeper ===
- Run state transitions by changing state in zookeeper rather than inside in Master.
+ Run state transitions by changing state in zookeeper rather than in Master.
  
- Keep up a region transition trail; regions move through states from unassigned to opening
to open, etc.  A region can't jump states as in going from unassigned to open.
+ Keep up a region transition trail; regions move through states from ''unassigned'' to ''opening''
to ''open'', etc.  A region can't jump states as in going from ''unassigned'' to ''open''.
+ 
+ Master (or client) moves regions between states.  Watchers on RegionServers notice additions
and act on it.  Master (or client) can do transitions in bulk; e.g. assign a regionserver
50 regions to open on startup.  Effect is that Master "pushes" work out to regionservers rather
than wait on them to heartbeat.
  
  A problem we have in current master is that states do not form a circle.  Once a region
is open, master stops keeping state; region state is moved to .META. table once assigned with
its condition checked periodically by .META. table scan.  Makes for confusion and evil such
as region double assignment because there are race condition potholes as we move from one
system -- internal state maps in master -- to the other during update to state in .META. 
Current thinking is to keep region lifecycle all up in zookeeper but that won't scale.  Postulate
100k regions -- 100TB at 1G regions -- each with two or three possible states each with watchers
for state change is too much to put in a zk cluster.  TODO: how to manage transition from
zk to .META.?
  
- <<Anchor(distinct)>>
- === In Zookeeper, a State and a Schema section ===
- Two locations in zk; one for schema and then one for state.  No connection.  For example,
could have hierarchy in zk as follows:
- 
- {{{/hbase/tables/name/schema/{family1, family2}
- /hbase/tables/name/state/{attributes [read-only, enabled, nocompact, noflush]}
- /hbase/regionservers/openregions/{list of regions...}
- /hbase/regionserver/to_open/{list of regions....}
- /hbase/regionservers/to_close/{list of regions...}
- 
  <<Anchor(clean)>>
- === State changes are clean, minimal, and comprehensive ===
+ === Region State changes are clean, minimal, and comprehensive ===
  Currently, moving a region from opening to open may involve a region compaction -- i.e.
a change to content in filesystem.  Better if modification of filesystem content was done
when no question of ownership involved.
  
  In current o.a.h.h.master.RegionManager.RegionState inner class, here are possible states:
@@ -87, +71 @@

      private volatile boolean offlined = false;}}}
  Its incomplete.
  
+ <<Anchor(state)>>
+ === Cluster/Table State ===
+ Move cluster and table state to zookeeper.
+ 
+ Do shutdown via ZK.
+ 
+ Remove table state change mechanism from master and instead update state in zk.  Clients
can set table state.  Watchers in regionservers react to table state changes.  This will make
it so table state changes are near instantaneous; e.g. setting table read-only, disabling
compactions/flushes and all without having to take table offline.
+ 
+ 
+ <<Anchor(distinct)>>
+ === In Zookeeper, a State and a Schema section ===
+ Two locations in zk; one for schema and then one for state.  No connection.  For example,
could have hierarchy in zk as follows:
+ 
+ {{{/hbase/tables/name/schema/{family1, family2}
+ /hbase/tables/name/state/{attributes [read-only, enabled, nocompact, noflush]}
+ /hbase/regionservers/openregions/{list of regions...}
+ /hbase/regionserver/to_open/{list of regions....}
+ /hbase/regionservers/to_close/{list of regions...}
+ 
+ 
  <<Anchor(schema)>>
  === Schema Edits ===
- Move Table Schema from .META.
+ Move Table Schema from .META.  Rather than storing complete Schema in each region, stored
once in ZK.
+ 
+ Edit Schema with tables online by making change in ZK.  Watchers inform watching RegionServers
of changes.  Make it so can add families, amend table and family configurations, etc.
  
  <<Anchor(balancer)>>
  === Region Assignment/Balancer ===
@@ -113, +119 @@

   * To Open Queue
    * Regionservers watch their own to open queues /hbase/rsopen/region(extra_info, which
hlogs to replay or it’s a split, etc)
  
+ Safe-mode assignment
+  * Collect all regions to assign
+  * Randomize and assign out in bulk, one msg per RS
+  * NO MORE SAFE-MODE
+  * Region assignment is always
+   * Look at all regions to be assigned
+   * Make a single decision for the assignment of all of these
+ 
  <<Anchor(root)>>
  === Remove -ROOT- ===
  Remove -ROOT- from filesystem; have it only live up in zk (Possible now Region Historian
feature has been removed).
@@ -135, +149 @@

  == Miscellaneous ==
  
   * At meetup we talked of moving .META. to zk and adding a getClosest to zk code base. 
Thats been punted on for now.
+  * At meetup we had design numbers but they don't make sense now we've lost the context
+      1. 200 regionservers
+      1. 32 outstanding wal logs per regionserver
+      1. 200 regions per regionserver being written to
+      1. 2GB or 30 hour full log roll
+      1. 10MB/sec write speed
+      1. 1.2M edits per 2G
+      1. 7k writes/second across cluster (?) -- whats this?  Wrong.
+      1. 1.2M edits per 30 hours?
+      1. 100 writes/sec across cluster (?) -- Whats this?  Wrong?
  
- 
- Administrative functions
-  * Hadoop RPC listeners on Master and Regionservers -- master can now push messages
-  * Clients and Master can talk to RS
- 
- Safe-mode assignment
-  * Collect all regions to assign
-  * Randomize and assign out in bulk, one msg per RS
-  * NO MORE SAFE-MODE
-  * Region assignment is always
-   * Look at all regions to be assigned
-   * Make a single decision for the assignment of all of these
- 
- 
- No more ROOT
- 
- Worker pool region closing
-  * Parallel flushes
- 
- No more CHANGE TABLE STATE
-  * Process server shutdown after RS crash
-   * Separate META scan?
-   * To figure out regions on an RS
-    * Separate map of RS -> region
-   * Trade-off between two-writes during assignment
- 
- Table Schema Information
-  * Online schema edits?
-   * If complex, punt to 0.22
-   * Rather than storing with each region, stored once in ZK
- 

Mime
View raw message