From common-commits-return-9788-apmail-hadoop-common-commits-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 20 19:36:40 2009 Return-Path: Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: (qmail 79902 invoked from network); 20 Oct 2009 19:36:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 20 Oct 2009 19:36:40 -0000 Received: (qmail 75665 invoked by uid 500); 20 Oct 2009 19:36:39 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 75617 invoked by uid 500); 20 Oct 2009 19:36:39 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 75608 invoked by uid 500); 20 Oct 2009 19:36:39 -0000 Delivered-To: apmail-hadoop-core-commits@hadoop.apache.org Received: (qmail 75605 invoked by uid 99); 20 Oct 2009 19:36:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 19:36:39 +0000 X-ASF-Spam-Status: No, hits=-2.5 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: apache.org Received: from [140.211.11.130] (HELO eos.apache.org) (140.211.11.130) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 19:36:35 +0000 Received: from eos.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id AC55F16E2D; Tue, 20 Oct 2009 19:36:15 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki To: Apache Wiki Date: Tue, 20 Oct 2009 19:36:15 -0000 Message-ID: <20091020193615.7580.39191@eos.apache.org> Subject: =?utf-8?q?=5BHadoop_Wiki=5D_Trivial_Update_of_=22Hbase/MasterRewrite=22_b?= =?utf-8?q?y_stack?= Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch= ange notification. The "Hbase/MasterRewrite" page has been changed by stack. http://wiki.apache.org/hadoop/Hbase/MasterRewrite?action=3Ddiff&rev1=3D7&re= v2=3D8 -------------------------------------------------- Initial Master Rewrite design came of conversations had at the hbase hack= athon held at StumbleUpon, August 5-7, 2009 ([[https://issues.apache.org/ji= ra/secure/attachment/12418561/HBase+Hackathon+Notes+-+Sunday.pdf|Jon Gray k= ept notes]]). The umbrella issue for the master rewrite is [[https://issue= s.apache.org/jira/browse/HBASE-1816|HBASE-1816]]. Timeline is hbase 0.20.1. = =3D=3D Table of Contents =3D=3D - * What does the Master do now? + * [[#now|What does the Master do now?]] - * Problems with current Master + * [[#problems|Problems with current Master]] - * Design - * = + * [[#design|Design]] + * [[#all|Move all region state transitions to zookeeper]] + * [[#distinct|In Zookeeper, a State and a Schema section]] = + <> =3D=3D What does the Master do now? =3D=3D Here's a bit of a refresher on what Master currently does: * Region Assignment @@ -22, +24 @@ * Distributes out administered close, flush, compact messages * Watches ZK for its own lease and for regionservers so knows when to ru= n recovery = + = + <> =3D=3D Problems with current Master =3D=3D There is a good list in the [[https://issues.apache.org/jira/secure/Manag= eLinks.jspa?id=3D12434794|Issue Links]] section of HBASE-1816. = - = + <> =3D=3D Design =3D=3D = + <> =3D=3D=3D Move all region state transitions to zookeeper =3D=3D=3D Run state transitions by changing state in zookeeper rather than inside i= n Master = + <> =3D=3D=3D In Zookeeper, a State and a Schema section =3D=3D=3D - = - =3D=3D Notes =3D=3D - To be organized... - = - =3D=3D=3D Schema vs State =3D=3D=3D - Chatting w/ Berk, Ryan, and J-D: - = - Two locations in zk; one for schema and then one for state. For example,= could have hierarchy in zk as follows: + Two locations in zk; one for schema and then one for state. No connectio= n. For example, could have hierarchy in zk as follows: = {{{/hbase/tables/name/schema/{family1, family2} /hbase/tables/name/state/{attributes [read-only, enabled, nocompact, nofl= ush]} @@ -47, +46 @@ /hbase/regionserver/to_open/{list of regions....} /hbase/regionservers/to_close/{list of regions...} = - Open of a region does not change state -- do not compact or rewrite anyth= ing -- in case when we go back to zk to report open, its been opened elsewh= ere. + =3D=3D=3D State changes are clean, minimal, and comprehensive =3D=3D=3D + Currently, moving a region from opening to open may involve a region comp= action -- i.e. a change to content in filesystem. Better if modification o= f filesystem content was done when no question of ownership involved. = - Need to come up with state transitions. Eliminate fuzzy state. + In current o.a.h.h.master.RegionManager.RegionState inner class, here are= possible states: + {{{ private volatile boolean unassigned =3D false; + private volatile boolean pendingOpen =3D false; + private volatile boolean open =3D false; + private volatile boolean closing =3D false; + private volatile boolean pendingClose =3D false; + private volatile boolean closed =3D false; + private volatile boolean offlined =3D false;}}} + = = =3D=3D More Notes =3D=3D =20