apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Shading Guava In Apex
Date Thu, 26 Nov 2015 11:07:13 GMT
One case where this is tricky is if any jars cause native libraries to be
loaded.  The problem is that you can't load these libraries more than once.
Furthermore, if you load the library in one non-root class loader, then the
other class loaders can access the native library. This make things really
complicated.



On Thu, Nov 26, 2015 at 10:47 AM, Thomas Weise <thomas@datatorrent.com>
wrote:

> I referred to the problem at Apex level that the platform developer faces.
>
> The same repeats, as you say, at the application level. Some operators
> depend on Hadoop directly, others share little or no dependencies.
>
>
> On Wed, Nov 25, 2015 at 3:28 PM, Chetan Narsude (cnarsude) <
> cnarsude@cisco.com> wrote:
>
> >
> > >But here, we depend on Hadoop so we cannot isolate it.
> >
> > User does not necessarily depend on Hadoop. The platform pulls the Hadoop
> > dependency into application code irrespective of user's influence. The
> > platform also pulls in other non-hadoop related N random dependencies
> from
> > application's perspective.
> >
> > The complexity comes from the effort needed to refactor parts of the code
> > to isolate the UDF and its dependencies from platform dependencies. It¹s
> a
> > very large initiative (that¹s where the motivation part comes) but not an
> > impossible one.
> >
> > Shading is a shady approach but it¹s an easy way out at smaller scale. At
> > larger scale it starts foraying into the pain around application
> > jar/platform jar deployment discussion and JVM PermGen memory issues.
> >
> > ‹
> > Chetan
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message