hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-13907) Document how to deploy a coprocessor
Date Tue, 14 Jul 2015 03:02:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625740#comment-14625740
] 

Sean Busbey edited comment on HBASE-13907 at 7/14/15 3:01 AM:
--------------------------------------------------------------

{code}
+ ... This behavior
+  is not expected to change. Alternately, you can include the coprocessor version in its
+  classname. This way you can unload the old coprocessor and load the new one,
+  without restarting the RegionServer. However, the old version will remain in memory,
+  so this solution is a temporary one, and restarting will eventually be necessary.
{code}

This currently works? Are we still loading additional jars at runtime for Coprocessors? Folks
who are running coprocessors are already kind of down a dark path, but should we encourage
the cleaner "rolling restart of region servers" to reduce the chances we have to debug classloader
pain?

{code}
+Coprocessor Logging::
+  The Coprocessor framework does not provide an API for logging beyond standard Java
+  logging. You can log to the location specified in the system property `HBASE_LOG_DIR`,
+  which is set in the RegionServer's `hbase-env.sh` file.
{code}

Won't using commons-logging or slf4j work? What do we gain by having them attempt to log directly
to this directory instead of just using the same logging framework we use?

{code}
+Coprocessor Configuration::
+  If you do not want to load coprocessors from the HBase Shell, you can add their configuration
+  properties to `hbase-site.xml`. In <<load_coprocessor_in_shell>>, two arguments
are
+  set: `arg1=1,arg2=2`. These could have been added to `hbase-site.xml` as follows:
{code}

just for my own clarity, if they use the hbase-site.xml approach then admins loading the coprocessor
in the shell won't be able to override the arguments, right? We should mention, either way.


was (Author: busbey):
{code}
+ ... This behavior
+  is not expected to change. Alternately, you can include the coprocessor version in its
+  classname. This way you can unload the old coprocessor and load the new one,
+  without restarting the RegionServer. However, the old version will remain in memory,
+  so this solution is a temporary one, and restarting will eventually be necessary.
{code}

This currently works? Are we still loading additional jars at runtime for Coprocessors? Folks
who are running coprocessors are already kind of down a dark path, but should we encourage
the cleaner "rolling restart of region servers" to reduce the changes we have to debug classloader
pain?

{code}
+Coprocessor Logging::
+  The Coprocessor framework does not provide an API for logging beyond standard Java
+  logging. You can log to the location specified in the system property `HBASE_LOG_DIR`,
+  which is set in the RegionServer's `hbase-env.sh` file.
{code}

Won't using commons-logging or slf4j work? What do we gain by having them attempt to log direclty
to this directory instead of just using the same logging framework we use?

{code}
+Coprocessor Configuration::
+  If you do not want to load coprocessors from the HBase Shell, you can add their configuration
+  properties to `hbase-site.xml`. In <<load_coprocessor_in_shell>>, two arguments
are
+  set: `arg1=1,arg2=2`. These could have been added to `hbase-site.xml` as follows:
{code}

just for my own clarity, if they use the hbase-site.xml approach then admins loading the coprocessor
in the shell won't be able to override the arguments, right? We should mention, either way.

> Document how to deploy a coprocessor
> ------------------------------------
>
>                 Key: HBASE-13907
>                 URL: https://issues.apache.org/jira/browse/HBASE-13907
>             Project: HBase
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Misty Stanley-Jones
>            Assignee: Misty Stanley-Jones
>         Attachments: HBASE-13907-1.patch, HBASE-13907-2.patch, HBASE-13907.patch
>
>
> Capture this information:
> > Where are the dependencies located for these classes? Is there a path on HDFS or
local disk that dependencies need to be placed so that each RegionServer has access to them?
> It is suggested to bundle them as a single jar so that RS can load the whole jar and
resolve dependencies. If you are not able to do that, you need place the dependencies in regionservers
class path so that they are loaded during RS startup. Do either of these options work for
you? Btw, you can load the coprocessors/filters into path specified by hbase.dynamic.jars.dir
[1], so that they are loaded dynamically by regionservers when the class is accessed (or you
can place them in the RS class path too, so that they are loaded during RS JVM startup).
> > How would one deploy these using an automated system? (puppet/chef/ansible/etc)
> You can probably use these tools to automate shipping the jars to above locations?
> > Tests our developers have done suggest that simply disabling a coprocessor, replacing
the jar with a different version, and enabling the coprocessor again does not load the newest
version. With that in mind how does one know which version is currently deployed and enabled
without resorting to parsing `hbase shell` output or restarting hbase?
> Actually this is a design issue with current classloader. You can't reload a class in
a JVM unless you delete all the current references to it. Since the current JVM (classloader)
has reference to it, you can't overwrite it unless you kill the JVM, which is equivalent to
restarting it. So you still have the older class loaded in place. For this to work, classloader
design should be changed. If it works for you, you can rename the coprocessor class name and
the new version of jar and RS loads it properly.
> > Where does logging go, and how does one access it? Does logging need to be configured
in a certain way?
> Can you please specify which logging you are referring to?
> > Where is a good location to place configuration files?
> Same as above, are these hbase configs or something else? If hbase configs, are these
gateway configs/server side? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message