flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishnu Viswanath <vishnu.viswanat...@gmail.com>
Subject Re: Class Cache
Date Mon, 31 Jul 2017 21:34:16 GMT
We have also noticed such behaviour when running on Yarn, had to restart
the session for the changes in the jar to be picked up.

On Mon, 31 Jul 2017 at 17:13, Ufuk Celebi <uce@apache.org> wrote:

> Hey Mike!
>
> Thanks for the detailed information about your setup. I'm also puzzled
> by this...
>
> (1) Which version of Flink are you using? We recently merged some
> changes to the JAR distribution components, which might cause this
> (although I think that's unlikely).
>
> (2) As a temporary work around you could try putting mylib2.jar into
> the /lib folder and not pulling it in via --classpath. After doing
> this you would need to stop/start the cluster and resubmit the job.
>
> – Ufuk
>
>
> On Mon, Jul 31, 2017 at 10:17 PM, Stephan Ewen <sewen@apache.org> wrote:
> > Hi Mike!
> >
> > Flink does in fact cache jar files in the "blob server". But these are
> > cached subject to the following conditions:
> >
> >   - No caching across "sessions", meaning start/stop of the
> > cluster/jobmanager. If you run the per-job-yarn setup, the job does not
> > cache anything.
> >
> >   - Files are cached under a content hash, meaning as soon as the
> contents
> > changes, the artifact is not reused. So if you actually change the jar
> > file, no caching should happen.
> >
> > I cannot really explain what you are observing and have never seen that
> > myself...
> >
> > Stephan
> >
> >
> > On Mon, Jul 31, 2017 at 9:00 PM, Mike Accola <maccola@us.ibm.com> wrote:
> >
> >> No, I did not explicitly  create an uber-jar.  The mylib1.jar is very
> >> light. It only contains my main application class (including
> >> ProcessFunction).
> >>
> >> I have been specifying --classpath option on my flink run command to
> pull
> >> in the mylib2.jar .
> >>
> >> Plus, I have been rebuilding mylib1.jar frequently just to be safe and
> it
> >> hasn't made a difference.
> >>
> >>
> >> Mike Accola
> >> maccola@us.ibm.com
> >>
> >>
> >>
> >>
> >>
> >> From:   Eron Wright <eronwright@gmail.com>
> >> To:     dev@flink.apache.org
> >> Date:   07/31/2017 01:47 PM
> >> Subject:        Re: Class Cache
> >>
> >>
> >>
> >> A Flink program is typically packaged as an 'uber-jar' containing its
> >> dependencies.  The Flink quickstart project illustrates this (see the
> use
> >> of the shading plugin in pom.xml).   Based on your description, the
> >> classes
> >> of mylib2.jar were copied into mylib1.jar when the latter was built.
> Try
> >> rebuilding mylib1.jar to effect the change.
> >>
> >> -Eron
> >>
> >> On Mon, Jul 31, 2017 at 11:18 AM, Mike Accola <maccola@us.ibm.com>
> wrote:
> >>
> >> > Are classes cached somewhere in flink?  I am running in a very basic,
> >> > local environment on Linux (start_local.sh).  I've somehow gotten my
> >> > environment into a strange state that I don't understand.  I feel
> like I
> >> > am overlooking something simple, but I've checked everything I can
> think
> >> > of.
> >> >
> >> > My main flink application with a ProcessFunction is embedded in
> >> > mylib1.jar.  Within my ProcessFunction I use another class that is
> >> > embedded in mylib2.jar.
> >> >
> >> > When I made changes to function in mylib2.jar and rebuilt the jar, I
> >> > realized the changes weren't taking affect.  In fact, I then delete
> >> > mylib2.jar entirely and my application still worked.  I can't figure
> out
> >> > where my application is picking up the function contained in
> mylib2.jar.
> >> I
> >> > have checked any temp directories, library paths, etc.  I have
> >> repeatedly
> >> > stopped/started my flink environment just to be safe.
> >> >
> >> > I tried adding -verbose:class to env.java.opts.  It output a lot of
> >> class
> >> > loading info to the stdout log, but there were no references to my
> class
> >> > in mylib2.jar.
> >> >
> >> > This has to be caching this code somehow whether it is in flink or in
> >> the
> >> > jvm.  Any ideas what could be happening or how to debug this further?
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message