asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raman Grover <ramangrove...@gmail.com>
Subject Re: External function dependency problem
Date Fri, 01 Jul 2016 17:11:28 GMT
Operations related to setting up an external library are contained in
ExternalLibraryUtil
<https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java>

At line 382, we have
// create and return the class loader

ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader);
return classLoader;

Above, we have the parentClassLoader set to the classloader for
ExternalLibraryUtil which is the application class loader (AsterixDB's
classloader that loads the dependencies from pom.xml). The proposed
solution (a) in earlier thread - skipping application classloader would
translate to replacing the above code with

ClassLoader classLoader = new URLClassLoader(urls, null);

Regards,
Raman

On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <xikuiw@uci.edu> wrote:

> >
> > Hi Abdullah,
>
> I reverted my code to reproduce the problem. Noticing this external
> function has a couple of other bugs but the dependency one is blocking
> others, so this should be enough to reproduce the problem.
>
> >
> The external function package is geoTag.zip.
>
> > ‚Äč
> >
> > ‚ÄčTest scripts are in tweetGeoTag.zip
>
> External function is loading data from data/, i.e.: data/state.json . So
> all json files in data.zip need to be placed under
> ../asterixdb/asterix-app/data/
>
> The real_tweets_adm.adm used in ddl is also attached.
>
> This setting will cause
>
> java.lang.NoSuchMethodError:
> > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
> > at
> > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541)
> > at
> > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452)
> > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17)
> > at
> >
> edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29)
> > at
> >
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101)
> > at
> >
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20)
> > at
> >
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18)
> > at
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > at
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > at scala.collection.immutable.List.foreach(List.scala:381)
> > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> > at scala.collection.immutable.List.map(List.scala:285)
> > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18)
> > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15)
> > at
> >
> edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34)
> > at
> >
> org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113)
> >
>
> The attachments exceed the size limits. I moved them under this google
> drive directory:
>
>
> https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing
>
> Let me know if you need anything else. Thanks.
>
> Best,
> Xikui
>
> On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <bamousaa@gmail.com>
> wrote:
>
> > I will look into that.
> >>
> >> @Xikui,
> >> Do you have a reproducable use case where such a behavior can be
> observed?
> >>
> >> Cheers,
> >> Abdullah.
> >>
> >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <ramangrover29@gmail.com>
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > Each external library has an separately associated custom class loader
> >> > (URLClassLoader) that loads the contained functions and any
> dependencies
> >> > (jar files) packaged inside the library. The custom loader fits into
> the
> >> > natural hierarchy with system (classpath based) classloader as the
> >> > immediate parent and bootstrap classloader as the root. As the
> >> delegation
> >> > model works, a class loader attempts to load a class only after
> >> parent(s)
> >> > in the lineage have failed.
> >> >
> >> > In your specific case, the system class loader is able to find the
> >> jackson
> >> > library from the configured classpath and so the library classloader
> is
> >> not
> >> > involved at all for this jar. The problem arising due to different
> >> versions
> >> > of a jar present across the hierarchy of classloaders is similar to
> one
> >> > faced in web containers such as Tomcat or application servers such as
> >> > Websphere where system loaded classes can override any application
> >> specific
> >> > classes due to name conflict.
> >> >
> >> > The way you are defining a library is correct. The fix needs to be in
> >> the
> >> > way the classes are loaded by AsterixDB (here acting as an application
> >> > server).
> >> >
> >> > I see two ways:-
> >> >
> >> > a) Skipping the system class loader: AsterixDB uses a URLClassloader
> to
> >> > load libraries. Setting the  parent of the this classloader as null
> >> implies
> >> > the bootrstrap classloader (root) as the immediate parent. The
> bootstrap
> >> > classloader loads rt.jar (java runtime classes) and i18n.jar and these
> >> > definitely do not have conflicts with your library dependencies (if
> they
> >> > do, not sure why one would override the classes in rt.jar and so we
> >> need to
> >> > fix the library). Note that the system classloader is out of the
> >> hierarchy
> >> > here and so any other jars loaded by AsterixDB from its classpath
> >> (pom.xml)
> >> > would not be visible.
> >> >
> >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a
> custom
> >> > implementation of classloader that attempts to load a class prior to
> >> > delegating to the parent. This way, the library classes and packaged
> >> > dependencies override any system level classes from the class path and
> >> even
> >> > the classes contained in rt.jar and i18n.jar.
> >> >
> >> > I am opening the discussion here to suggest further alternatives or
> >> provide
> >> > preferences.
> >> >
> >> > I have a preference for (a)  (skipping the system class loader) for
> two
> >> > reasons:
> >> >
> >> > a) it is simpler
> >> >
> >> > b) the other option allows a custom class loader to override classes
> in
> >> > rt.jar., which is ok but not how classloaders are supposed to work in
> >> > principle.
> >> >
> >> > Regards,
> >> >
> >> > Raman
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <dtabass@gmail.com> wrote:
> >> >
> >> > > Any classloader experts have suggestions...?
> >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <xikuiw@uci.edu> wrote:
> >> > >
> >> > > > Hi Devs,
> >> > > >
> >> > > > We found a problem when trying to build external functions for
the
> >> > > > cloudberry demo.
> >> > > >
> >> > > > When the external function is depend on certain library, the
> library
> >> > that
> >> > > > comes with the external function will be blocked by same library
> in
> >> > > > AsterixDB. In our case, our external function 'geoTag' uses
> jackson
> >> > > v2.7.1,
> >> > > > and we packed all dependencies into one single jar. When running
> >> > 'geoTag'
> >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes
> >> > > > NullPointerException. We have to manually change pom.xml in
> >> AsterixDB
> >> > to
> >> > > > fix that.
> >> > > >
> >> > > > We are wondering is that because we load the external function
in
> a
> >> > wrong
> >> > > > way, or this could be one possible interesting problem which
is
> >> worth
> >> > > > noticing. Thanks.
> >> > > >
> >> > > > Best,
> >> > > > Xikui
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>



-- 
Raman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message