Return-Path: Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: (qmail 44737 invoked from network); 30 Sep 2010 18:49:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 30 Sep 2010 18:49:39 -0000 Received: (qmail 3495 invoked by uid 500); 30 Sep 2010 18:49:39 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 3355 invoked by uid 500); 30 Sep 2010 18:49:38 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 3347 invoked by uid 99); 30 Sep 2010 18:49:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 18:49:38 +0000 X-ASF-Spam-Status: No, hits=1.8 required=10.0 tests=FH_HELO_EQ_D_D_D_D,MIME_QP_LONG_LINE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [184.73.217.71] (HELO ip-10-202-7-187.ec2.internal) (184.73.217.71) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 18:49:31 +0000 Received: from ip-10-202-7-187.ec2.internal (localhost [127.0.0.1]) by ip-10-202-7-187.ec2.internal (Postfix) with ESMTP id C10BC8A1FE; Thu, 30 Sep 2010 18:49:10 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions From: "Andrew Purtell" To: "Andrew Purtell" , "Jonathan Gray" , "Mingjie Lai" , jiraposter@review.hbase.org, dev@hbase.apache.org Date: Thu, 30 Sep 2010 18:49:10 -0000 Message-ID: <20100930184910.12012.10882@ip-10-202-7-187.ec2.internal> In-Reply-To: <20100930181739.12040.88236@ip-10-202-7-187.ec2.internal> References: <20100930181739.12040.88236@ip-10-202-7-187.ec2.internal> X-Virus-Checked: Checked by ClamAV on apache.org > On 2010-09-30 11:17:39, Jonathan Gray wrote: > > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2= 999 > > > > > > I would actually be -1 on having this call the get coprocessor hook= s. Expected behavior (for me) would be that coprocessors hook into the cli= ent-side calls, not internal calls. For checkAndPut, are those Gets also w= rapped by coprocessors? > > = > > I suppose you could make an argument either way, but I'd err on the= side of coprocessors attaching on client operations not internal ones. We don't wrap checkAndPut, we wrap the component Get and Put. Likewise with= ICV. No problem to change to strictly wrap client operations. Will address= in an updated patch. - Andrew ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/876/#review1366 ----------------------------------------------------------- On 2010-09-19 18:35:17, Mingjie Lai wrote: > = > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > http://review.cloudera.org/r/876/ > ----------------------------------------------------------- > = > (Updated 2010-09-19 18:35:17) > = > = > Review request for hbase. > = > = > Summary > ------- > = > = > The diff actually contains 2 seperate patches: HBase-2001 and the one for= (HBASE-2002+HBASE-2321). The reason is that HBase-2001's CommandTarget rel= ies on HBASE-2002 + HBASE-2321 which patches are still under review. I have= to include Gary's HBASE-2002, HBASE-2321 with this diff, since reviewboard= is so powerful :) and it disallow my diff to be based on some unchecked in= patch. = > = > Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total nu= mber of lines are more than 7k. I turned back and forth, but still don't ha= ve a good idea to create the patch in order to reduce the review pain. Howe= ver right now I'm putting the whole patch for all the 3 issues. Here the li= st of file which are only related to coprocessor: > = > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java > src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.= java > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.ja= va > src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java > src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java > src/main/resources/hbase-default.xml > src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java > src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java > src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterfac= e.java > src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInter= face.java > src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStack= ing.java > = > = > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > = > (Here is a brief description. Please find much more details at the packag= e-info.java in the diff. I also post the package-info.html to https://issue= s.apache.org/jira/browse/HBASE-2001 as an attachment.) > = > = > Coprocessors are code that runs in-process on each region server. Regions= contain references to the coprocessor implementation classes associated wi= th them. Coprocessor classes will be loaded either from local jars on the r= egion server's classpath or via the HDFS classloader. > = > Multiple types of coprocessors are provided to provide sufficient flexibi= lity for potential use cases. Right now there are: > = > * Coprocessor: provides region lifecycle management hooks, e.g., region o= pen/close/split/flush/compact operations. > * RegionObserver: provides hook for monitor table operations from client = side, such as table get/put/scan/delete, etc. > * CommandTarget: provides on demand triggers for any arbitrary function e= xecuted at a region. One use case is column aggregation at region server. > = > Coprocessor: > A coprocessor is required to implement Coprocessor interface so that copr= ocessor framework can manage it internally. > = > Another design goal of this interface is to provide simple features for m= aking coprocessors useful, while exposing no more internal state or control= actions of the region server than necessary and not exposing them directly= . = > = > RegionObserver > If the coprocessor implements the RegionObserver interface it can observe= and mediate client actions on the region. = > = > CommandTarget: > Coprocessor and RegionObserver provide certain hooks for injecting user c= ode running at each region. These code will be triggerd with existing HTabl= e and HBaseAdmin operations at the certain hook points. > = > Through CommandTarget and dynamic RPC protocol, you can define your own i= nterface communicated between client and region server, i.e., you can speci= fy new passed parameters and return types for a method. And the new Command= Target methods can be triggered by calling client side dynamic RPC function= s -- HTable.exec(...). = > = > Coprocess loading > A customized coprocessor can be loaded by two different ways, by configur= ation, or by HTableDescriptor for a newly created table. > = > (Currently we don't really have an on demand coprocessor loading machanis= m for opened regions. However it should be easy to create a dedicated Comma= ndTarget for coprocessor loading) = > = > = > This addresses bug HBase-2001. > http://issues.apache.org/jira/browse/HBase-2001 > = > = > Diffs > ----- > = > src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81 = > src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATI= ON = > src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618 = > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java fb= dec0b = > src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263 = > src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838 = > src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b = > src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0 = > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.jav= a PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.ja= va PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-= CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironmen= t.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.= java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java P= RE-CREATION = > src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE= -CREATION = > src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f62= 3d = > src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CRE= ATION = > src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-= CREATION = > src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREAT= ION = > src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6 = > src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629 = > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d = > src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java a4810a6 = > src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CR= EATION = > src/main/java/org/apache/hadoop/hbase/master/HMaster.java bccdc0e = > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java= PRE-CREATION = > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java fdef130 = > src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f= 2e4e7c = > src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.jav= a bb3b382 = > src/main/resources/hbase-default.xml 5452fd1 = > src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java= PRE-CREATION = > src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.jav= a PRE-CREATION = > src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterf= ace.java PRE-CREATION = > src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInt= erface.java PRE-CREATION = > src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverSta= cking.java PRE-CREATION = > src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProt= ocol.java PRE-CREATION = > = > Diff: http://review.cloudera.org/r/876/diff > = > = > Testing > ------- > = > = > Thanks, > = > Mingjie > = >