flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Frontend classpath issue
Date Fri, 24 Feb 2017 10:05:34 GMT
Hi Robert,

I was not aware of this big change (I know it's my fault) but I am not sure
if I agree with the rationale.

I read through the JIRA and it seems that this is mostly a convenience
change that we dont need to copy jars and mess with the classloading that
much.

On the other hand if user jars can conflict with frontend/backend classes
that can lead to very serious (and hard to fix) problems, especially in
larger scale deployments.

What do you think about this?

Gyula

Robert Metzger <rmetzger@apache.org> ezt írta (időpont: 2017. febr. 23.,
Cs, 22:10):

> Mh. The user jar is put into every classpath. So the jobmanager /
> taskmanagers are potentially affected by this as well.
> Probably the data transfer between the TMs doesn't call the same methods
> as the UI on the JobManager :)
>
> The simplest solution is to shade your netty in the user jar into a
> different location.
>
>
> On Thu, Feb 23, 2017 at 10:01 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:
>
> Hi Robert,
> It definitely explains the behaviour.
>
> This only applies to the frontend right?
> If so what is the rationale behind it, and how should I handle the
> dependency conflict?
>
> Thanks,
> Gyula
>
> Robert Metzger <rmetzger@apache.org> ezt írta (időpont: 2017. febr. 23.,
> Cs, 21:44):
>
> Hi,
> Since Flink 1.2 "per job yarn applications" (when you do "-m
> yarn-cluster") include the job jar into the classpath as well.
> Does this change explain the behavior?
>
> On Thu, Feb 23, 2017 at 4:59 PM, Gyula Fóra <gyfora@apache.org> wrote:
>
> Hi,
>
> I have a problem that the frontend somehow seems to have the user jar on
> the classpath and it leads to a netty conflict:
>
> https://gist.github.com/gyfora/4ec2c8a8a6b33adb80d411460432ce8d
>
> So in the jobmanager logs I can see that my job started (running on YARN),
> but can't access the frontend, it gives internal server error with the
> previous exception. So I dont have the same jar problem on the actual
> running job.
>
> I haven't really seen this before, is this something that happened to
> somebody else as well?
>
> Thank you!
> Gyula
>
>
>
>

Mime
View raw message