hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Helmling (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-10169) Batch coprocessor
Date Wed, 05 Feb 2014 06:06:13 GMT

     [ https://issues.apache.org/jira/browse/HBASE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Gary Helmling updated HBASE-10169:

    Attachment: HBASE-10169-alternate.patch

After the review I posted on the reviewboard request last week, I went through the patch and
made a number of changes in the direction I had suggested.  My primary comments on the v5
patch were:

Thanks for all the work on this updated patch.  The new client APIs seem reasonable, though
I think they could use a minor simplification.

My biggest feedback is that MultiRegionServerCallable and RegionServerCoprocessorRpcInvoker
seem to duplicate a lot of what is provided by the existing MultiServerCallable and AsyncProcess.
 When I originally looked at making this same change in a previous version of HBase, I looked
at extending the Action and MultiAction classes to support coprocessor endpoint invocations
so that we could leverage the existing RPC mechanisms.  I still think that would be a better
approach here, rather that creating a parallel set of RPC classes specifically for coprocessor
endpoints.  But if you disagree, please just explain what problems you see with this approach.

Attached is a modified patch (HBASE-10169-alternate.patch), which modifies the code to use
AsyncProcess as suggested.  The main differences from the v5 patch are:

* In HTable batchCoprocessorService methods, replaces ServiceDescriptor + String method name
with MethodDescriptor
* Removes MultiRegionServerCallable and RegionServerCoprocessorRpcInvoker, using AsyncProcess
* Removes HRegionServer.execMultiService() method
* Removed the additional thread pool in HRegionServer.  If this is useful to have, we could
add it back in in a separate issue (and consider doing so for other batch operations at the
same time).
* Adds support for CoprocessorServiceCall to Action/MultiAction, and CoprocessorServiceResult
to MultiResponse/ResultOrException

[~jingcheng.du@intel.com], please look through the alternate patch and let me know if this
approach might work for you, or if you still see some needed functionality missing.

Note: TestBatchCoprocessorEndpoint#testAggregationWithNullResponse seems to be failing occasionally.
 I'm still trying to track that down.

> Batch coprocessor
> -----------------
>                 Key: HBASE-10169
>                 URL: https://issues.apache.org/jira/browse/HBASE-10169
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Coprocessors
>    Affects Versions: 0.99.0
>            Reporter: Jingcheng Du
>            Assignee: Jingcheng Du
>         Attachments: Batch Coprocessor Design Document.docx, HBASE-10169-V2.patch, HBASE-10169-V3.patch,
HBASE-10169-V3.patch, HBASE-10169-V4.patch, HBASE-10169-V5.patch, HBASE-10169-alternate.patch,
> This is designed to improve the coprocessor invocation in the client side. 
> Currently the coprocessor invocation is to send a call to each region. If there’s one
region server, and 100 regions are located in this server, each coprocessor invocation will
send 100 calls, each call uses a single thread in the client side. The threads will run out
soon when the coprocessor invocations are heavy. 
> In this design, all the calls to the same region server will be grouped into one in a
single coprocessor invocation. This call will be spread into each region in the server side.

This message was sent by Atlassian JIRA

View raw message