Return-Path: Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: (qmail 38112 invoked from network); 19 Nov 2010 22:39:15 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Nov 2010 22:39:15 -0000 Received: (qmail 42453 invoked by uid 500); 19 Nov 2010 22:39:46 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 42392 invoked by uid 500); 19 Nov 2010 22:39:46 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 42376 invoked by uid 99); 19 Nov 2010 22:39:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 22:39:46 +0000 X-ASF-Spam-Status: No, hits=4.0 required=10.0 tests=FH_HELO_EQ_D_D_D_D,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 184.73.217.71 is neither permitted nor denied by domain of mjlai09@gmail.com) Received: from [184.73.217.71] (HELO ip-10-202-7-187.ec2.internal) (184.73.217.71) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 22:39:39 +0000 Received: from ip-10-202-7-187.ec2.internal (localhost [127.0.0.1]) by ip-10-202-7-187.ec2.internal (Postfix) with ESMTP id 702BE8A20A; Fri, 19 Nov 2010 22:39:18 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions From: "Mingjie Lai" To: "Andrew Purtell" , "Jonathan Gray" , stack@duboce.net Date: Fri, 19 Nov 2010 22:39:18 -0000 Message-ID: <20101119223918.10414.75831@ip-10-202-7-187.ec2.internal> Cc: "Himanshu Vashishtha" , "Lars Francke" , "Mingjie Lai" , dev@hbase.apache.org, jiraposter@review.hbase.org, "Gary Helmling" In-Reply-To: <20101108224141.9239.80907@ip-10-202-7-187.ec2.internal> References: <20101108224141.9239.80907@ip-10-202-7-187.ec2.internal> X-Virus-Checked: Checked by ClamAV on apache.org ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/876/ ----------------------------------------------------------- (Updated 2010-11-19 14:39:18.378849) Review request for hbase, stack, Andrew Purtell, and Jonathan Gray. Changes ------- Final patch, ready to be checked in: - Truncated white space at line end - Rebuilt the patch after HBase-2002 checked in. = Summary ------- The diff actually contains 2 seperate patches: HBase-2001 and the one for (= HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget relie= s on HBASE-2002 + HBASE-2321 which patches are still under review. I have t= o include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard i= s so powerful :) and it disallow my diff to be based on some unchecked in p= atch. = Eventually the patch here should be committed after 2001 and 2321. I will m= ake another patch after they got checked in. = Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total numb= er of lines are more than 7k. I turned back and forth, but still don't have= a good idea to create the patch in order to reduce the review pain. Howeve= r right now I'm putting the whole patch for all the 3 issues. Here the list= of file which are only related to coprocessor: src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.j= ava src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoproce= ssor.java src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.ja= va src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint= .java src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtocol= .java src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.j= ava src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.= java src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterfa= ce.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStackin= g.java =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D (Here is a brief description. Please find much more details at the package-= info.java in the diff. I also post the package-info.html to https://issues.= apache.org/jira/browse/HBASE-2001 as an attachment.) Coprocessors are code that runs in-process on each region server. Regions c= ontain references to the coprocessor implementation classes associated with= them. Coprocessor classes will be loaded either from local jars on the reg= ion server's classpath or via the HDFS classloader. Multiple types of coprocessors are provided to provide sufficient flexibili= ty for potential use cases. Right now there are: * Coprocessor: provides region lifecycle management hooks, e.g., region ope= n/close/split/flush/compact operations. * RegionObserver: provides hook for monitor table operations from client si= de, such as table get/put/scan/delete, etc. * Endpoint: provides on demand triggers for any arbitrary function executed= at a region. One use case is column aggregation at region server. Coprocessor: A coprocessor is required to implement Coprocessor interface so that coproc= essor framework can manage it internally. Another design goal of this interface is to provide simple features for mak= ing coprocessors useful, while exposing no more internal state or control a= ctions of the region server than necessary and not exposing them directly. = RegionObserver If the coprocessor implements the RegionObserver interface it can observe a= nd mediate client actions on the region. = Endpoint: Coprocessor and RegionObserver provide certain hooks for injecting user cod= e running at each region. These code will be triggerd with existing HTable = and HBaseAdmin operations at the certain hook points. Through Endpoint and dynamic RPC protocol, you can define your own interfac= e communicated between client and region server, i.e., you can create a new= method, specify passed parameters and return types for the method. And the= new Endpoint methods can be triggered by calling client side dynamic RPC f= unctions -- HTable.exec(...). = Coprocess loading A customized coprocessor can be loaded by two different ways, by configurat= ion, or by HTableDescriptor for a newly created table. (Currently we don't really have an on demand coprocessor loading machanism = for opened regions. However it should be easy to create a dedicated Command= Target for coprocessor loading) = This addresses bug HBase-2001. http://issues.apache.org/jira/browse/HBase-2001 Diffs (updated) ----- src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor= .java PRE-CREATION = src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCopro= cessor.java PRE-CREATION = src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CR= EATION = src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.= java PRE-CREATION = src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.ja= va PRE-CREATION = src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE= -CREATION = src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-C= REATION = src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java P= RE-CREATION = src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java d669de1 = src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 2ef= 6432 = src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.j= ava 1309f93 = src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java = 1bcde8c = src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHand= ler.java ace7997 = src/main/resources/hbase-default.xml 630c041 = src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoi= nt.java PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationProtoc= ol.java PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.ja= va PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint= .java PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterfac= e.java PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInter= face.java PRE-CREATION = src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStack= ing.java PRE-CREATION = Diff: http://review.cloudera.org/r/876/diff Testing ------- Thanks, Mingjie