hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrian Muraru (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7205) Coprocessor classloader is replicated for all regions in the HRegionServer
Date Sun, 09 Dec 2012 01:35:22 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527317#comment-13527317

Adrian Muraru commented on HBASE-7205:

Right, classloading java is tough, I agree 
Regarding your comments:
The reason why the above assertion failed was that the classloader for jarFileOnHDFS2 was
removed from classLoadersCache in the middle of the test because of attempt of loading cpNameInvalid
True, *cpNameInvalid* fails to load (no such class in cp jar) however the same jar (i.e. its
associated classloader) manages to successfully loads *another coprocessor: cpName2*
Take a closer look on how the *test* table is created in {{TestClassLoading#testClassLoadingFromHDFS}}:
    htd.addFamily(new HColumnDescriptor("test"));
      // without configuration values
    htd.setValue("COPROCESSOR$1", jarFileOnHDFS1.toString() + "|" + cpName1 + "|" + Coprocessor.PRIORITY_USER);
      // with configuration values
    htd.setValue("COPROCESSOR$2", jarFileOnHDFS2.toString() + "|" + cpName2 + "|" + Coprocessor.PRIORITY_USER
+ "|k1=v1,k2=v2,k3=v3");
    // invalid class name (should fail to load this class)
    htd.setValue("COPROCESSOR$3", jarFileOnHDFS2.toString() + "|" + cpNameInvalid + "|" +
See, the same jar file {{jarFileOnHDFS2}} is used to load two different coprocessor classes
(one is successfully loaded, the other not). 
What should we do in this case? 
My take is to keep the classloader in cache and allow other regions to re-use it.
That's the reason I removed classloaderCache.remove() and I strongly go for it.

Now, the fundamental question : 
*Should we silently ignore failures in CP loading (excepting the warn message in log) ?*
I think we should be more restrictive, propagate the failures upstream to the table handler
and fail to bring the HRegion online in this case.
What do you think?

I think the above assertion places extra limit on how CoprocessorHost.load() handles ClassNotFoundException.
It assumes that the classloader corresponding to attempt of loading invalid classname (more
strictly, classname and jar file mismatch) would be retained in cache.
No, that's not true: The assertion is checking that *all region active-classloaders (i.e.
those that managed to successfully load at least one CP) are all cached*

> Coprocessor classloader is replicated for all regions in the HRegionServer
> --------------------------------------------------------------------------
>                 Key: HBASE-7205
>                 URL: https://issues.apache.org/jira/browse/HBASE-7205
>             Project: HBase
>          Issue Type: Bug
>          Components: Coprocessors
>    Affects Versions: 0.92.2, 0.94.2
>            Reporter: Adrian Muraru
>            Assignee: Ted Yu
>            Priority: Critical
>             Fix For: 0.96.0, 0.94.4
>         Attachments: 7205-v1.txt, 7205-v3.txt, 7205-v4.txt, 7205-v5.txt, 7205-v6.txt,
7205-v7.txt, 7205-v8.txt, HBASE-7205_v2.patch
> HBASE-6308 introduced a new custom CoprocessorClassLoader to load the coprocessor classes
and a new instance of this CL is created for each single HRegion opened. This leads to OOME-PermGen
when the number of regions go above hundres / region server. 
> Having the table coprocessor jailed in a separate classloader is good however we should
create only one for all regions of a table in each HRS.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message