asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raman Grover <ramangrove...@gmail.com>
Subject Re: External function dependency problem
Date Sun, 03 Jul 2016 02:23:46 GMT
i am missing the attachment
On Jul 2, 2016 6:39 PM, "Xikui Wang" <xikuiw@uci.edu> wrote:

> Hi Raman,
>
> Thanks for your help. I tried this quick fix on my branch, but it
> introduces some new exceptions. I think this causes Asterix fails at
> entering the external function. The error message is attached.
>
> Best,
> Xikui
>
> On Fri, Jul 1, 2016 at 10:11 AM, Raman Grover <ramangrover29@gmail.com>
> wrote:
>
>> Operations related to setting up an external library are contained in
>> ExternalLibraryUtil
>> <
>> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
>> >
>>
>> At line 382, we have
>> // create and return the class loader
>>
>> ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader);
>> return classLoader;
>>
>> Above, we have the parentClassLoader set to the classloader for
>> ExternalLibraryUtil which is the application class loader (AsterixDB's
>> classloader that loads the dependencies from pom.xml). The proposed
>> solution (a) in earlier thread - skipping application classloader would
>> translate to replacing the above code with
>>
>> ClassLoader classLoader = new URLClassLoader(urls, null);
>>
>> Regards,
>> Raman
>>
>> On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <xikuiw@uci.edu> wrote:
>>
>> > >
>> > > Hi Abdullah,
>> >
>> > I reverted my code to reproduce the problem. Noticing this external
>> > function has a couple of other bugs but the dependency one is blocking
>> > others, so this should be enough to reproduce the problem.
>> >
>> > >
>> > The external function package is geoTag.zip.
>> >
>> > > ‚Äč
>> > >
>> > > ‚ÄčTest scripts are in tweetGeoTag.zip
>> >
>> > External function is loading data from data/, i.e.: data/state.json . So
>> > all json files in data.zip need to be placed under
>> > ../asterixdb/asterix-app/data/
>> >
>> > The real_tweets_adm.adm used in ddl is also attached.
>> >
>> > This setting will cause
>> >
>> > java.lang.NoSuchMethodError:
>> > > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
>> > > at
>> > >
>> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541)
>> > > at
>> > >
>> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452)
>> > > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17)
>> > > at
>> > >
>> >
>> edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29)
>> > > at
>> > >
>> >
>> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101)
>> > > at
>> > >
>> >
>> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20)
>> > > at
>> > >
>> >
>> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18)
>> > > at
>> > >
>> >
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>> > > at
>> > >
>> >
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>> > > at scala.collection.immutable.List.foreach(List.scala:381)
>> > > at
>> scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>> > > at scala.collection.immutable.List.map(List.scala:285)
>> > > at
>> edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18)
>> > > at
>> edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15)
>> > > at
>> > >
>> >
>> edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34)
>> > > at
>> > >
>> >
>> org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113)
>> > >
>> >
>> > The attachments exceed the size limits. I moved them under this google
>> > drive directory:
>> >
>> >
>> >
>> https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing
>> >
>> > Let me know if you need anything else. Thanks.
>> >
>> > Best,
>> > Xikui
>> >
>> > On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <bamousaa@gmail.com
>> >
>> > wrote:
>> >
>> > > I will look into that.
>> > >>
>> > >> @Xikui,
>> > >> Do you have a reproducable use case where such a behavior can be
>> > observed?
>> > >>
>> > >> Cheers,
>> > >> Abdullah.
>> > >>
>> > >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <
>> ramangrover29@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Hi
>> > >> >
>> > >> > Each external library has an separately associated custom class
>> loader
>> > >> > (URLClassLoader) that loads the contained functions and any
>> > dependencies
>> > >> > (jar files) packaged inside the library. The custom loader fits
>> into
>> > the
>> > >> > natural hierarchy with system (classpath based) classloader as
the
>> > >> > immediate parent and bootstrap classloader as the root. As the
>> > >> delegation
>> > >> > model works, a class loader attempts to load a class only after
>> > >> parent(s)
>> > >> > in the lineage have failed.
>> > >> >
>> > >> > In your specific case, the system class loader is able to find
the
>> > >> jackson
>> > >> > library from the configured classpath and so the library
>> classloader
>> > is
>> > >> not
>> > >> > involved at all for this jar. The problem arising due to different
>> > >> versions
>> > >> > of a jar present across the hierarchy of classloaders is similar
to
>> > one
>> > >> > faced in web containers such as Tomcat or application servers
such
>> as
>> > >> > Websphere where system loaded classes can override any application
>> > >> specific
>> > >> > classes due to name conflict.
>> > >> >
>> > >> > The way you are defining a library is correct. The fix needs to
be
>> in
>> > >> the
>> > >> > way the classes are loaded by AsterixDB (here acting as an
>> application
>> > >> > server).
>> > >> >
>> > >> > I see two ways:-
>> > >> >
>> > >> > a) Skipping the system class loader: AsterixDB uses a
>> URLClassloader
>> > to
>> > >> > load libraries. Setting the  parent of the this classloader as
null
>> > >> implies
>> > >> > the bootrstrap classloader (root) as the immediate parent. The
>> > bootstrap
>> > >> > classloader loads rt.jar (java runtime classes) and i18n.jar and
>> these
>> > >> > definitely do not have conflicts with your library dependencies
(if
>> > they
>> > >> > do, not sure why one would override the classes in rt.jar and
so we
>> > >> need to
>> > >> > fix the library). Note that the system classloader is out of the
>> > >> hierarchy
>> > >> > here and so any other jars loaded by AsterixDB from its classpath
>> > >> (pom.xml)
>> > >> > would not be visible.
>> > >> >
>> > >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide
a
>> > custom
>> > >> > implementation of classloader that attempts to load a class prior
>> to
>> > >> > delegating to the parent. This way, the library classes and
>> packaged
>> > >> > dependencies override any system level classes from the class
path
>> and
>> > >> even
>> > >> > the classes contained in rt.jar and i18n.jar.
>> > >> >
>> > >> > I am opening the discussion here to suggest further alternatives
or
>> > >> provide
>> > >> > preferences.
>> > >> >
>> > >> > I have a preference for (a)  (skipping the system class loader)
for
>> > two
>> > >> > reasons:
>> > >> >
>> > >> > a) it is simpler
>> > >> >
>> > >> > b) the other option allows a custom class loader to override
>> classes
>> > in
>> > >> > rt.jar., which is ok but not how classloaders are supposed to
work
>> in
>> > >> > principle.
>> > >> >
>> > >> > Regards,
>> > >> >
>> > >> > Raman
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <dtabass@gmail.com>
wrote:
>> > >> >
>> > >> > > Any classloader experts have suggestions...?
>> > >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <xikuiw@uci.edu>
wrote:
>> > >> > >
>> > >> > > > Hi Devs,
>> > >> > > >
>> > >> > > > We found a problem when trying to build external functions
for
>> the
>> > >> > > > cloudberry demo.
>> > >> > > >
>> > >> > > > When the external function is depend on certain library,
the
>> > library
>> > >> > that
>> > >> > > > comes with the external function will be blocked by
same
>> library
>> > in
>> > >> > > > AsterixDB. In our case, our external function 'geoTag'
uses
>> > jackson
>> > >> > > v2.7.1,
>> > >> > > > and we packed all dependencies into one single jar.
When
>> running
>> > >> > 'geoTag'
>> > >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB
which
>> causes
>> > >> > > > NullPointerException. We have to manually change pom.xml
in
>> > >> AsterixDB
>> > >> > to
>> > >> > > > fix that.
>> > >> > > >
>> > >> > > > We are wondering is that because we load the external
function
>> in
>> > a
>> > >> > wrong
>> > >> > > > way, or this could be one possible interesting problem
which is
>> > >> worth
>> > >> > > > noticing. Thanks.
>> > >> > > >
>> > >> > > > Best,
>> > >> > > > Xikui
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Raman
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message