hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell" <apurt...@apache.org>
Subject Re: Review Request: HBase-2001: Coprocessors: Colocate user code with regions
Date Sun, 26 Sep 2010 00:10:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/876/#review1323
-----------------------------------------------------------



src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
<http://review.cloudera.org/r/876/#comment4433>

    Please add javadoc that explains that coprocessors loaded from configuration are all installed
at System priority, chained in the order they appear in the value. (So users must explicitly
order authoritative extensions before observers.)



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4462>

    If this throws a CoprocessorException the read lock won't be unlocked. Move out of the
finally block. 



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4463>

    This is not really post flush. Should be moved after internalFlushcache()



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4464>

    What happened to checkResources()?



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4466>

    Spelling error



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4467>

    Why not? What is excluded?



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4465>

    Spelling error



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/876/#comment4468>

    Yes we do. The coprocessor may be intercepting gets and performing global substitutions.
Not allowing it to do so in this case would introduce inconsistency.



src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
<http://review.cloudera.org/r/876/#comment4469>

    What are these magic numbers?


- Andrew


On 2010-09-19 18:35:17, Mingjie Lai wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/876/
> -----------------------------------------------------------
> 
> (Updated 2010-09-19 18:35:17)
> 
> 
> Review request for hbase.
> 
> 
> Summary
> -------
> 
> 
> The diff actually contains 2 seperate patches: HBase-2001 and the one for (HBASE-2002+HBASE-2321).
The reason is that HBase-2001's CommandTarget relies on HBASE-2002 + HBASE-2321 which patches
are still under review. I have to include Gary's HBASE-2002, HBASE-2321 with this diff, since
reviewboard is so powerful :) and it disallow my diff to be based on some unchecked in patch.

> 
> Both HBase-2001 and the dynamic RPC stuff are quite big patches. Total number of lines
are more than 7k. I turned back and forth, but still don't have a good idea to create the
patch in order to reduce the review pain. However right now I'm putting the whole patch for
all the 3 issues. Here the list of file which are only related to coprocessor:
> 
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
> src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
> src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> src/main/resources/hbase-default.xml
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
> src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
> 
> 
> ==========================
> 
> (Here is a brief description. Please find much more details at the package-info.java
in the diff. I also post the package-info.html to https://issues.apache.org/jira/browse/HBASE-2001
as an attachment.)
> 
> 
> Coprocessors are code that runs in-process on each region server. Regions contain references
to the coprocessor implementation classes associated with them. Coprocessor classes will be
loaded either from local jars on the region server's classpath or via the HDFS classloader.
> 
> Multiple types of coprocessors are provided to provide sufficient flexibility for potential
use cases. Right now there are:
> 
> * Coprocessor: provides region lifecycle management hooks, e.g., region open/close/split/flush/compact
operations.
> * RegionObserver: provides hook for monitor table operations from client side, such as
table get/put/scan/delete, etc.
> * CommandTarget: provides on demand triggers for any arbitrary function executed at a
region. One use case is column aggregation at region server.
> 
> Coprocessor:
> A coprocessor is required to implement Coprocessor interface so that coprocessor framework
can manage it internally.
> 
> Another design goal of this interface is to provide simple features for making coprocessors
useful, while exposing no more internal state or control actions of the region server than
necessary and not exposing them directly. 
> 
> RegionObserver
> If the coprocessor implements the RegionObserver interface it can observe and mediate
client actions on the region. 
> 
> CommandTarget:
> Coprocessor and RegionObserver provide certain hooks for injecting user code running
at each region. These code will be triggerd with existing HTable and HBaseAdmin operations
at the certain hook points.
> 
> Through CommandTarget and dynamic RPC protocol, you can define your own interface communicated
between client and region server, i.e., you can specify new passed parameters and return types
for a method. And the new CommandTarget methods can be triggered by calling client side dynamic
RPC functions -- HTable.exec(...). 
> 
> Coprocess loading
> A customized coprocessor can be loaded by two different ways, by configuration, or by
HTableDescriptor for a newly created table.
> 
> (Currently we don't really have an on demand coprocessor loading machanism for opened
regions. However it should be easy to create a dedicated CommandTarget for coprocessor loading)

> 
> 
> This addresses bug HBase-2001.
>     http://issues.apache.org/jira/browse/HBase-2001
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/hadoop/hbase/client/Action.java 556ea81 
>   src/main/java/org/apache/hadoop/hbase/client/Exec.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/client/ExecResult.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/client/HConnection.java 65f7618 
>   src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java fbdec0b 
>   src/main/java/org/apache/hadoop/hbase/client/HTable.java 0dbf263 
>   src/main/java/org/apache/hadoop/hbase/client/MultiAction.java c6ea838 
>   src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java 91bd04b 
>   src/main/java/org/apache/hadoop/hbase/client/RowRange.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/client/Scan.java 29b3cb0 
>   src/main/java/org/apache/hadoop/hbase/coprocessor/BaseCommandTarget.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorException.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 83f623d 
>   src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorProtocol.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java e23a629 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java e4c356d 
>   src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java a4810a6 
>   src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/master/HMaster.java bccdc0e 
>   src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java fdef130 
>   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f2e4e7c 
>   src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bb3b382 
>   src/main/resources/hbase-default.xml 5452fd1 
>   src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassloading.java PRE-CREATION

>   src/test/java/org/apache/hadoop/hbase/coprocessor/TestCommandTarget.java PRE-CREATION

>   src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java PRE-CREATION

>   src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
PRE-CREATION 
>   src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java PRE-CREATION

>   src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java PRE-CREATION

> 
> Diff: http://review.cloudera.org/r/876/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Mingjie
> 
>


Mime
View raw message