hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-16264) Figure how to deal with endpoints and shaded pb
Date Mon, 29 Aug 2016 19:19:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438171#comment-15438171
] 

stack edited comment on HBASE-16264 at 8/29/16 7:19 PM:
--------------------------------------------------------

Interesting. 'True', exterior endpoints are packaged in an Endpoint jar. When we load the
endpoint, it triggers the coprocessor classloader. If the endpoint makes use of hbase internal
classes, i.e. if the endpoint overrides hbase classes that it wants the classloader prefer
over what is in hbase (e.g. the generated protobuf classes for HRegionInfo or Cell, in this
case, versions of these classes that used unshaded protobuf), then there is a mechanism for
forcing the classloader to prefer the classes in the endpoint jar rather than defer to the
parent classloader; in the Endpoint configuration, pass a list of classes keyed by HConstants.CP_HTD_ATTR_INCLUSION_KEY
and noted classes will be classloaded by the Endpoint classloader rather than the parent classloader.
Nice.

None of this works though for the bundled endpoints, endpoints whose code lives in the midst
of the rest of our code; e.g. MultiRowMutationEndpoint or AccessController. These are intermixed
with other hbase code in the hbase-server module. I can't break them out of the module because
they depend on core hbase classes. Because they are intermixed, they do not trigger the Endpoint
classloader.

I messed around and was able to have the Endpoint classloader trigger whenever one of our
bundled Endpoint classes was up for loading. The Classloader tries to load the CP 'environment'.
At this point, down in the generated protobuf, we do some Class.forName loading. Though the
Endpoint Classloader is set as preferred on the thread, its the default classloader that gets
used down in the generated code (generated code would have to seek out the thread Classloader
explicitly). The default classloader has already loaded ClientProtos etc., from the shaded
module; I can't intercept the classloading, not unless I replace the system classloader.

Let me try a different tact (after chatting w/ [~busbey]). I'll put in place an intercepting
classloader on the server. Anytime it sees com.google.protobuf, it will replace relocate the
class to point at the shaded version. Let me see how this does.


was (Author: stack):
Interesting. For true exterior endpoints, the are packaged in an endpoint jar. When we load
the endpoint, it triggers the coprocessor classloader. If the endpoint makes use of hbase
internal classes -- i.e. if the endpoint overrides hbase classes that it wants the classloader
prefer over what is in hbase (e.g. the generated protobuf classes for HRegionInfo or Cell,
in this case, versions of these classes that used unshaded protobuf) -- then there is also
a mechanism for forcing the classloader to prefer the classes in the endpoint jar rather than
defer to the parent classloader by passing a list of classes keyed by HConstants.CP_HTD_ATTR_INCLUSION_KEY
as coprocessor attributes. Nice.

None of this works though for the bundled endpoints; MultiRowMutationEndpoint or AccessController.
These are intermixed with other hbase code in the hbase-server module. I can't break them
out of the module because they depend on hbase classes to do their work. So, because they
are intermixed, I can't selectively do classloading based off origin jar/module.

> Figure how to deal with endpoints and shaded pb
> -----------------------------------------------
>
>                 Key: HBASE-16264
>                 URL: https://issues.apache.org/jira/browse/HBASE-16264
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Coprocessors, Protobufs
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HBASE-16264.master.001.patch, HBASE-16264.master.002.patch, HBASE-16264.master.003.patch,
HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, HBASE-16264.master.006.patch,
HBASE-16264.master.007.patch
>
>
> Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. Would be
sweet if could make it so all just worked. At worst, come up w/ a prescription for how to
migrate existing CPs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message