hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2001) Coprocessors: Colocate user code with regions
Date Fri, 04 Jun 2010 04:55:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875476#action_12875476
] 

HBase Review Board commented on HBASE-2001:
-------------------------------------------

Message from: "Andrew Purtell" <apurtell@apache.org>


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java, line 140
bq.  > <http://review.hbase.org/r/96/diff/4/?file=892#file892line140>
bq.  >
bq.  >     Which table is this?

Any table. The idea is the coprocessor can create any private tables it needs to implement
its functionality.


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java, line 27
bq.  > <http://review.hbase.org/r/96/diff/4/?file=892#file892line27>
bq.  >
bq.  >     What about unloading?  You remember that conversation up on irc of how loading
is one thing but unloading w/o breakage is hard prob.

I'm not trying to tackle unloading yet.

However, classes are strongly bound to their classloader. We do instantiate a classloader
each time to load a coprocessor and we don't hold a reference to the classloader. It is my
understanding that when there are no more references to the classes (no live objects), they
and the classloader will be garbage collected, though the JVM spec does not guarantee this
the Sun JVM will do this. Creating a new classloader and asking for the class again, presumably
from an updated jar, should load the new class -- a unit test can verify. 

To help insure old classes don't leak via live objects hanging around, we could consider a
cooperative lifecycle management scheme like that used by OSGi: http://en.wikipedia.org/wiki/OSGi.



bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java, line 150
bq.  > <http://review.hbase.org/r/96/diff/4/?file=892#file892line150>
bq.  >
bq.  >     Needs to be Writable?

No.


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java, line 29
bq.  > <http://review.hbase.org/r/96/diff/4/?file=893#file893line29>
bq.  >
bq.  >     This is a mixin you'd use if you want to be notified about compactions, etc.?

This is for translating values found in a flush file into new values in the new storefile
being built by the compaction, or for dropping values. 


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 85
bq.  > <http://review.hbase.org/r/96/diff/4/?file=894#file894line85>
bq.  >
bq.  >     What would this be used for?  For CP to call out elsewhere on a table?

The idea is the coprocessor can create any private tables it needs to implement its functionality.
But we want to mediate that, add access control, clean up references when/if the cp is terminated
(and perhaps unloaded).


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 237
bq.  > <http://review.hbase.org/r/96/diff/4/?file=894#file894line237>
bq.  >
bq.  >     Is this needed?

Why not.


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 245
bq.  > <http://review.hbase.org/r/96/diff/4/?file=895#file895line245>
bq.  >
bq.  >     Its always on then?

Yes, otherwise I have to wrap all calls to the cp host in HRegion with "if (coprocessorHost
!= null) then", including the inner loops of the major and minor compactors. 


bq.  On 2010-06-02 23:28:50, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2885
bq.  > <http://review.hbase.org/r/96/diff/4/?file=895#file895line2885>
bq.  >
bq.  >     Who would want to get at this?

Tests. So probably this does not have to be public.


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/96/#review123
-----------------------------------------------------------





> Coprocessors: Colocate user code with regions
> ---------------------------------------------
>
>                 Key: HBASE-2001
>                 URL: https://issues.apache.org/jira/browse/HBASE-2001
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>         Attachments: asm-3.2-bin.zip, asm-transformations.pdf, HBASE-2001-RegionObserver.patch,
HBASE-2001.patch.gz
>
>
> Support user code that runs run next to each region in table. As regions split and move,
coprocessor code should automatically  move also.
> Use classloader which looks on HDFS.
> Associate a list of classes to load with each table. Put this in HRI so it inherits from
table but can be changed on a per region basis (so then those region specific changes can
inherited by daughters). 
> Not completely arbitrary code, should require implementation of an interface with callbacks
for:
> * Open
> * Close
> * Split
> * Compact
> * (Multi)get and scanner next()
> * (Multi)put
> * (Multi)delete
> Add method to HRegionInterface for invoking coprocessor methods and retrieving results.
 
> Add methods in o.a.h.h.regionserver or subpackage which implement convenience functions
for coprocessor methods and consistent/controlled access to internals: store access, threading,
persistent and ephemeral state, scratch storage, etc. 
> GitHub: http://github.com/apurtell/hbase-coprocessor

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message