asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raman Grover <ramangrove...@gmail.com>
Subject Re: External function dependency problem
Date Thu, 30 Jun 2016 16:53:00 GMT
Hi

Each external library has an separately associated custom class loader
(URLClassLoader) that loads the contained functions and any dependencies
(jar files) packaged inside the library. The custom loader fits into the
natural hierarchy with system (classpath based) classloader as the
immediate parent and bootstrap classloader as the root. As the delegation
model works, a class loader attempts to load a class only after parent(s)
in the lineage have failed.

In your specific case, the system class loader is able to find the jackson
library from the configured classpath and so the library classloader is not
involved at all for this jar. The problem arising due to different versions
of a jar present across the hierarchy of classloaders is similar to one
faced in web containers such as Tomcat or application servers such as
Websphere where system loaded classes can override any application specific
classes due to name conflict.

The way you are defining a library is correct. The fix needs to be in the
way the classes are loaded by AsterixDB (here acting as an application
server).

I see two ways:-

a) Skipping the system class loader: AsterixDB uses a URLClassloader to
load libraries. Setting the  parent of the this classloader as null implies
the bootrstrap classloader (root) as the immediate parent. The bootstrap
classloader loads rt.jar (java runtime classes) and i18n.jar and these
definitely do not have conflicts with your library dependencies (if they
do, not sure why one would override the classes in rt.jar and so we need to
fix the library). Note that the system classloader is out of the hierarchy
here and so any other jars loaded by AsterixDB from its classpath (pom.xml)
would not be visible.

b) Defining a Custom Class loader: Inside AsterixDB, we provide a custom
implementation of classloader that attempts to load a class prior to
delegating to the parent. This way, the library classes and packaged
dependencies override any system level classes from the class path and even
the classes contained in rt.jar and i18n.jar.

I am opening the discussion here to suggest further alternatives or provide
preferences.

I have a preference for (a)  (skipping the system class loader) for two
reasons:

a) it is simpler

b) the other option allows a custom class loader to override classes in
rt.jar., which is ok but not how classloaders are supposed to work in
principle.

Regards,

Raman







On Jun 29, 2016 11:07 PM, "Mike Carey" <dtabass@gmail.com> wrote:

> Any classloader experts have suggestions...?
> On Jun 29, 2016 10:26 PM, "Xikui Wang" <xikuiw@uci.edu> wrote:
>
> > Hi Devs,
> >
> > We found a problem when trying to build external functions for the
> > cloudberry demo.
> >
> > When the external function is depend on certain library, the library that
> > comes with the external function will be blocked by same library in
> > AsterixDB. In our case, our external function 'geoTag' uses jackson
> v2.7.1,
> > and we packed all dependencies into one single jar. When running 'geoTag'
> > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes
> > NullPointerException. We have to manually change pom.xml in AsterixDB to
> > fix that.
> >
> > We are wondering is that because we load the external function in a wrong
> > way, or this could be one possible interesting problem which is worth
> > noticing. Thanks.
> >
> > Best,
> > Xikui
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message