apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Shading Guava In Apex
Date Wed, 25 Nov 2015 23:19:39 GMT
I guess the topic is how to use different guava in same JVM,(for this
purpose simply shade the class is good enough)

*NOT* how to generally support isolation of different library dependencies
for different apps.

BTW I think an isolated container sounds like a hadoop feature (
https://issues.apache.org/jira/browse/HADOOP-10893)

On Wed, Nov 25, 2015 at 3:06 PM, Thomas Weise <thomas@datatorrent.com>
wrote:

> Chetan,
>
> Thanks for writing this up.
>
> Apex depends on the Hadoop client (with it's version of dependency X). If
> Apex changes X to another version and that version isn't backward
> compatible, then the Hadoop client is potentially broken.
>
> Trying to solve this with class loaders can be tricky. One approach is to
> completely isolate class loaders. In some systems that works (EJB or
> servlet
> containers). But here, we depend on Hadoop so we cannot isolate it.
>
> So what if we really really need a newer version of X? That's where shading
> helps. We don't touch whatever Hadoop was certified to work with. Instead,
> we repackage X and change the import statements in our code that depends on
> it so there is no conflict with the rest of the stack. That's what we had
> to do with ASM, I guess this is why Flink shaded guava.
>
> Thomas
>
> On Wed, Nov 25, 2015 at 2:35 PM, Chetan Narsude (cnarsude) <
> cnarsude@cisco.com> wrote:
>
> > Continuing this thread to elaborate on what I meant:
> >
> > When an application developer writes Apex application, and they use
> binary
> > incompatible version of jar (using extreme case to simplify elaboration)
> > than that Hadoop depends on, depending upon how the class path is defined
> > - JVM will either load the version that application wants or the one that
> > Hadoop wants and one of the two entities are destined to malfunction. I
> > faced this situation with an early Apex customer incidentally with the
> > same jar under discussion - Guava and had no way out. Incidentally
> > Cloudera¹s revision to their distro included the version of Guava we
> > wanted and conflict went away.
> >
> > Another flavor is (which we have fixed both in cli and gateway now) is
> > when cli is used to launch application1 which has a class a.b.c.classname
> > and later without terminating the cli   is used to launch application2
> > which requires a different class which incidentally uses the same package
> > and classname - application2 is launched with the class which was meant
> > for application1. Although this is more likely to happen with dependency
> > version changes, the first time I encountered - I was making  changes to
> > the code, recompiling, and creating a new jar.
> >
> > Servlet Containers apparently solve this problem because they have to
> load
> > multiple applications within their own dependency (possibly conflicting)
> > in the same JVM space. They do some black magic (it seems trivial though)
> > with ClassLoaders. With Apex we do not do any of this magic (due to tight
> > coupling among Hadoop/Apex/Application) so the entire stack is vulnerable
> > to class leaks.
> >
> > It can be solved in theory and has been proven by Servlet Containers but
> I
> > hear one more person saying (Ted - correct me if I mis-interpret) that
> > it¹s a steep effort.
> >
> > ‹
> > Chetan
> >
> >
> >
> > On 11/25/15, 12:31 PM, "Ted Dunning" <ted.dunning@gmail.com> wrote:
> >
> > >On Thu, Nov 26, 2015 at 4:15 AM, Timothy Farkas <tim@datatorrent.com>
> > >wrote:
> > >
> > >> I thought frameworks such as OSGi are used to isolate classes.
> > >>Application
> > >> servers like GlassFish use that technique to isolate application
> > >> dependencies from platform dependencies. Cask has also implemented a
> > >> similar technique for their platform
> > >>
> > >
> > >I have tried to use OSGI in the past. It was mind bogglingly difficult
> to
> > >get right. My sense is that OSGI has decreased in popularity quite a lot
> > >in
> > >recent years.  Glassfish is hardly gaining adherents at this point.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message