asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xikui Wang <xik...@uci.edu>
Subject Re: External function dependency problem
Date Thu, 30 Jun 2016 23:41:13 GMT
>
> Hi Abdullah,

I reverted my code to reproduce the problem. Noticing this external
function has a couple of other bugs but the dependency one is blocking
others, so this should be enough to reproduce the problem.

>
The external function package is geoTag.zip.

> ‚Äč
>
> ‚ÄčTest scripts are in tweetGeoTag.zip

External function is loading data from data/, i.e.: data/state.json . So
all json files in data.zip need to be placed under
../asterixdb/asterix-app/data/

The real_tweets_adm.adm used in ddl is also attached.

This setting will cause

java.lang.NoSuchMethodError:
> com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
> at
> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541)
> at
> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452)
> at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17)
> at
> edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29)
> at
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101)
> at
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20)
> at
> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> at scala.collection.immutable.List.map(List.scala:285)
> at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18)
> at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15)
> at
> edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34)
> at
> org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113)
>

The attachments exceed the size limits. I moved them under this google
drive directory:

https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing

Let me know if you need anything else. Thanks.

Best,
Xikui

On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <bamousaa@gmail.com>
wrote:

> I will look into that.
>>
>> @Xikui,
>> Do you have a reproducable use case where such a behavior can be observed?
>>
>> Cheers,
>> Abdullah.
>>
>> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <ramangrover29@gmail.com>
>> wrote:
>>
>> > Hi
>> >
>> > Each external library has an separately associated custom class loader
>> > (URLClassLoader) that loads the contained functions and any dependencies
>> > (jar files) packaged inside the library. The custom loader fits into the
>> > natural hierarchy with system (classpath based) classloader as the
>> > immediate parent and bootstrap classloader as the root. As the
>> delegation
>> > model works, a class loader attempts to load a class only after
>> parent(s)
>> > in the lineage have failed.
>> >
>> > In your specific case, the system class loader is able to find the
>> jackson
>> > library from the configured classpath and so the library classloader is
>> not
>> > involved at all for this jar. The problem arising due to different
>> versions
>> > of a jar present across the hierarchy of classloaders is similar to one
>> > faced in web containers such as Tomcat or application servers such as
>> > Websphere where system loaded classes can override any application
>> specific
>> > classes due to name conflict.
>> >
>> > The way you are defining a library is correct. The fix needs to be in
>> the
>> > way the classes are loaded by AsterixDB (here acting as an application
>> > server).
>> >
>> > I see two ways:-
>> >
>> > a) Skipping the system class loader: AsterixDB uses a URLClassloader to
>> > load libraries. Setting the  parent of the this classloader as null
>> implies
>> > the bootrstrap classloader (root) as the immediate parent. The bootstrap
>> > classloader loads rt.jar (java runtime classes) and i18n.jar and these
>> > definitely do not have conflicts with your library dependencies (if they
>> > do, not sure why one would override the classes in rt.jar and so we
>> need to
>> > fix the library). Note that the system classloader is out of the
>> hierarchy
>> > here and so any other jars loaded by AsterixDB from its classpath
>> (pom.xml)
>> > would not be visible.
>> >
>> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a custom
>> > implementation of classloader that attempts to load a class prior to
>> > delegating to the parent. This way, the library classes and packaged
>> > dependencies override any system level classes from the class path and
>> even
>> > the classes contained in rt.jar and i18n.jar.
>> >
>> > I am opening the discussion here to suggest further alternatives or
>> provide
>> > preferences.
>> >
>> > I have a preference for (a)  (skipping the system class loader) for two
>> > reasons:
>> >
>> > a) it is simpler
>> >
>> > b) the other option allows a custom class loader to override classes in
>> > rt.jar., which is ok but not how classloaders are supposed to work in
>> > principle.
>> >
>> > Regards,
>> >
>> > Raman
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Jun 29, 2016 11:07 PM, "Mike Carey" <dtabass@gmail.com> wrote:
>> >
>> > > Any classloader experts have suggestions...?
>> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <xikuiw@uci.edu> wrote:
>> > >
>> > > > Hi Devs,
>> > > >
>> > > > We found a problem when trying to build external functions for the
>> > > > cloudberry demo.
>> > > >
>> > > > When the external function is depend on certain library, the library
>> > that
>> > > > comes with the external function will be blocked by same library in
>> > > > AsterixDB. In our case, our external function 'geoTag' uses jackson
>> > > v2.7.1,
>> > > > and we packed all dependencies into one single jar. When running
>> > 'geoTag'
>> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes
>> > > > NullPointerException. We have to manually change pom.xml in
>> AsterixDB
>> > to
>> > > > fix that.
>> > > >
>> > > > We are wondering is that because we load the external function in
a
>> > wrong
>> > > > way, or this could be one possible interesting problem which is
>> worth
>> > > > noticing. Thanks.
>> > > >
>> > > > Best,
>> > > > Xikui
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message