flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Budzyn <michal.budzyn.ext...@zalando.de>
Subject Re: Class loading and job versioning
Date Wed, 20 Jul 2016 15:18:35 GMT
Hi Stephan,

IMO the platform needs better jobs isolation. All this shading shouldn't 
be required at all.

Michal




On 20.07.2016 16:18, Stephan Ewen wrote:
> Hi Michael!
>
> The only safe way in Java to isolate the user code from the platform 
> would be to completely run them in different JVMs.
>
> Other than that, Max's method to ensure the correct instantiation 
> should work in most cases.
>
> We also continuously try to have fewer dependencies in Flink, to that 
> there is less to clash.
>
> Stephan
>
>
> On Wed, Jul 20, 2016 at 3:53 PM, Maximilian Michels <mxm@apache.org 
> <mailto:mxm@apache.org>> wrote:
>
>     Sure. No Problem.
>
>     The issue is a bit more involved. You're right, the user classes have
>     precedence over the Flink classpath. So your classes were probably
>     loaded fine. However, the user code also calls Flink code which can
>     use a library version different from the job jar library. And boom, it
>     crashes because the job jar library has been loaded previously :)
>
>     The only way we could circumvent this problem would be to explicitly
>     set a different classloader and load Flink internal classes via
>     reflection. I think the performance penalty for this would be way too
>     high.
>
>     Best,
>     Max
>
>     On Wed, Jul 20, 2016 at 2:23 PM, Michal Budzyn
>     <michal.budzyn.extern@zalando.de
>     <mailto:michal.budzyn.extern@zalando.de>> wrote:
>     > Thanks for the prompt replay.
>     >
>     > You are right. The conflict was between
>     "com.fasterxml.jackson.core" libs.
>     >
>     > I am just wondering. If the the jobs were separted from the
>     platform,
>     >
>     > the jobs libs should have precedence and no versioning problem
>     should have
>     > happened.
>     >
>     > Regards,
>     >
>     > Michal
>     >
>     >
>     >
>     > On 20.07.2016 14:00, Maximilian Michels wrote:
>     >>
>     >> Hi Michal,
>     >>
>     >> I couldn't find Joda in flink-dist. Possibly there is some
>     other clash?
>     >>
>     >> There are two potential issues here:
>     >>
>     >> 1) Flink shades some libraries (Guava) but not all. If you use a
>     >> version of a library in your Flink job which doesn't match the
>     one in
>     >> flink-dist, you're bound for trouble.
>     >>
>     >> 2) Flink separates jobs from each other to avoid potential class
>     >> version mismatches. Each job has its own classloader. In this
>     sense,
>     >> "job versioning" is supported.
>     >>
>     >> Cheers,
>     >> Max
>     >>
>     >> On Wed, Jul 20, 2016 at 1:02 PM, Michal Budzyn
>     >> <michal.budzyn.extern@zalando.de
>     <mailto:michal.budzyn.extern@zalando.de>> wrote:
>     >>>
>     >>> Hi all,
>     >>> We had a class versioning problem within Flink Job.
>     >>> The job uses Joda 2.6, but the flink-dist 1.0.3 packages 2.5.
>     >>> The problem was solved by relocating job classes with shade
>     plug-in.
>     >>>
>     >>> Does flink separate jobs from each other to avoid class
>     conflicts between
>     >>> them and the platform ?
>     >>> Is job versioning supported or is shading always required ?
>     >>>
>     >>> Regards,
>     >>> Michal
>     >>>
>     >
>
>


Mime
View raw message