flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Testing Apache Flink 0.9.0-rc1
Date Fri, 12 Jun 2015 09:09:04 GMT
Just to clarify. If you write a Flink program and include the Table API as
a dependency, then you have to package your program in the JAR with the
Table API and submit it to the cluster. IMHO that's ok but it should be
documented to inform users which libraries are included in Flink binaries
out of the box. Let's document and postpone a discussion whether Flink
should ship all libraries (and all their dependencies, as Till pointed out)
in the future. I think this is too big of a change for a release candidate.

On Fri, Jun 12, 2015 at 10:44 AM, Till Rohrmann <trohrmann@apache.org>
wrote:

> Yes you're right Ufuk. At the moment the user has to place the jars in the
> lib folder of Flink. If this folder is not shared then he has to do it for
> every node on which Flink runs.
>
> On Fri, Jun 12, 2015 at 10:42 AM Till Rohrmann <trohrmann@apache.org>
> wrote:
>
> > I think I found a real release blocker. Currently we don't add license
> > files to our shaded jars. For example
> > the flink-shaded-include-yarn-0.9.0-milestone-1.jar shades hadoop code.
> > This code also includes the `org.apache.util.bloom.*` classes. These
> > classes are licensed under  The European Commission project OneLab. We
> have
> > a notice in the LICENSE file of our binary distribution but I think we
> also
> > have to add them in the shaded jar. There might even be more code bundled
> > as part of some shaded jars which I have not spotted yet.
> >
> > Furthermore, I noticed that we list all Apache License dependencies in
> our
> > LICENSE file of our binary distribution (which we don't have to do).
> > However, we don't do it in our jars which contain for example guava and
> asm
> > as shaded dependencies. Maybe we should be consistent here.
> >
> > But maybe I overlook something here and we don't have to do it.
> >
> > On Fri, Jun 12, 2015 at 10:29 AM Till Rohrmann <trohrmann@apache.org>
> > wrote:
> >
> >> Well I think the initial idea was to keep the dist jar as small a
> >> possible and therefore we did not include the libraries. I'm not sure
> >> whether we can decide this here ad-hoc. If the community says that we
> shall
> >> include these libraries then I can add them. But bear in mind that all
> of
> >> them have some transitive dependencies which will be added as well.
> >>
> >>
> >> On Fri, Jun 12, 2015 at 10:15 AM Márton Balassi <
> balassi.marton@gmail.com>
> >> wrote:
> >>
> >>> As for outstanding issues I think streaming is good to go as far as I
> >>> know.
> >>> I am personally against including all libraries - at least speaking for
> >>> the
> >>> streaming connectors. Robert, Stephan and myself had a detailed
> >>> discussion
> >>> on that some time ago and the disadvantage of having all the libraries
> in
> >>> the distribution is the dependency mess that they pull. In this case I
> >>> would rather add documentation on putting them in the user jar then. As
> >>> for
> >>> the other libraries they do not depend on so much external code, so +1
> >>> for
> >>> putting them in.
> >>>
> >>> On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mxm@apache.org>
> >>> wrote:
> >>>
> >>> > We should have a nightly cluster test for every library. Let's keep
> >>> that in
> >>> > mind for the future. Very nice find, Till!
> >>> >
> >>> > Since there were not objections, I cherry-picked the proposed commits
> >>> from
> >>> > the document to the release-0.9 branch. If I understand correctly,
we
> >>> can
> >>> > create the new release candidate once Till has checked the licenses,
> >>> Ufuk's
> >>> > TableInput fix has been merged, and Fabian's web interface
> improvement
> >>> are
> >>> > in. Plus, we need to include all Flink libraries in flink-dist. Are
> you
> >>> > going to fix that as well, Till?
> >>> >
> >>> > On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uce@apache.org>
wrote:
> >>> >
> >>> > >
> >>> > > On 12 Jun 2015, at 09:45, Till Rohrmann <trohrmann@apache.org>
> >>> wrote:
> >>> > >
> >>> > > > Hi guys,
> >>> > > >
> >>> > > > I just noticed while testing the TableAPI on the cluster
that it
> >>> is not
> >>> > > > part of the dist module. Therefore, programs using the TableAPI
> >>> will
> >>> > only
> >>> > > > run when you put the TableAPI jar directly on the cluster
or if
> you
> >>> > > build a
> >>> > > > fat jar including the TableAPI jar. This is nowhere documented.
> >>> > > > Furthermore, this also applies to Gelly and FlinkML.
> >>> > >
> >>> > > I think all of these should be included in the fat jar. They are
> all
> >>> > > highly advertized components.
> >>> > >
> >>> > > Very good catch, Till! I didn't get around to testing Table API
on
> a
> >>> > > cluster, yet.
> >>> >
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message