Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGWx-_vh4KmcZe2zLPD=gpkySO8wsb36_QJZJVz7UfshabPv9Q@mail.gmail.com>
References: 
 <CANC1h_u6qtEsF1WCcoU1d38JGd+XTAQWmvp9Stx4vfe68BOjBw@mail.gmail.com>
 <CAKADb_No5y1h3b1zeQPsRaKrtHtx+KAR=jYBJAhJGz=RtWP=fQ@mail.gmail.com>
 <560D0EA2.9020002@apache.org>
 <CAGWx-_ufCdiXbCM44ymB9XmH3j42Xz=9uav1GkW3xy6htCX31w@mail.gmail.com>
 <CAGr9p8A3V0HtLSCHBd3xj3KC5bTSPTv-=DQbccy1UGjMc-YG=w@mail.gmail.com>
 <CANC1h_vDb3uTPF39XJCpo34ZjBChpsM3R_78JDXmeb5ThMASwg@mail.gmail.com>
 <560D34FD.60503@apache.org>
 <CAGWx-_vh4KmcZe2zLPD=gpkySO8wsb36_QJZJVz7UfshabPv9Q@mail.gmail.com>
From: Robert Metzger <rmetzger@apache.org>
Date: Thu, 1 Oct 2015 16:48:19 +0200
Message-ID: 
 <CAGr9p8C_oMhW+hOxzyThoLQaM=SPqFs5c5q_sNwz8JrWgaiYCw@mail.gmail.com>
Subject: Re: Pulling Streaming out of staging and project restructure
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a1135f0b283ee0305210c2480

--001a1135f0b283ee0305210c2480
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

@Chesnay: Nothing prevents projects from getting stuck there. But at least
we don't have two staging repositories (dist, staging). Also I would not
say that there has been no graduation out of staging. Yarn was also there
once, streaming and gelly are currently leaving it. So I would actually say
it worked.

One more thing: Can we actually put the web submission client into a
separate module until we have something new and better?

It does really not fit into "flink-clients". It makes Flink users pull the
jetty dependencies even if they never use them for example when running
Flink from an IDE.


On Thu, Oct 1, 2015 at 4:45 PM, Kostas Tzoumas <ktzoumas@apache.org> wrote:

> +1 to Robert and practicality :-)
>
> As I said before, I do not feel strongly about this, I was torn myself.
>
> On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
> > If we remove flink-staging because projects tend to get stuck there, wh=
at
> > mechanism prevents the same happening with flink-contrib?
> >
> >
> > On 01.10.2015 15:19, Stephan Ewen wrote:
> >
> >> +1 for Robert's comments.
> >>
> >> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetzger@apache.org>
> >> wrote:
> >>
> >> Big +1 for graduating streaming out of staging. It is widely used, als=
o
> in
> >>> production and we are spending a lot of effort into hardening it.
> >>> I also agree with the proposed new maven module structure.
> >>>
> >>> We have to carefully test the reworked structure for the scripts whic=
h
> >>> are
> >>> generating the hadoop1 and the scala 2.11 poms (they are transformed
> >>> using
> >>> a bunch of bash scripts). I can do that once the PR is open.
> >>>
> >>> @Chesnay: I would be fine with including the language binding into
> python
> >>>
> >>>> Where would new projects reside in, that previously would have been
> put
> >>>>
> >>> into flink-staging?
> >>>
> >>> flink-contrib
> >>>
> >>>
> >>> @Kostas: I understand the idea behind your suggested renaming, but
> thats
> >>> just a name. I don't think its going to influence how people are seei=
ng
> >>> Flink: It doesn't feel like second class when adding
> >>> "flink-streaming-core"
> >>> to the dependencies to me.
> >>> Also, the "flink-datastream-scala" module would depend on
> >>> "flink-dataset-scala", which is kind of weird.
> >>>
> >>>
> >>> I'm wondering whether we should remove the "flink-test-utils" module.=
 I
> >>> don't think its really necessary, because we can put the test jars in=
to
> >>> the
> >>> flink-tests project and include them using the "test-jar" dependency.
> >>>
> >>>
> >>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzoumas@apache.org>
> >>> wrote:
> >>>
> >>> +1
> >>>>
> >>>> I wanted to suggest that we rename modules to fully accept streaming
> as
> >>>> first class, qualifying also "batch" as "batch" (e.g., flink-java --=
>
> >>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >>>>
> >>>> This would break maven dependencies (temporary hell :-) so it's not =
a
> >>>> decision to take lightly. I'm not strongly advocating for it.
> >>>>
> >>>>
> >>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.or=
g
> >
> >>>> wrote:
> >>>>
> >>>> I like it in general. But while we're at it, what is the purpose of
> the
> >>>>> flink-tests project, or rather which tests belong there instead of
> the
> >>>>> individual projects?
> >>>>>
> >>>>> Where would new projects reside in, that previously would have been
> put
> >>>>> into flink-staging?
> >>>>>
> >>>>> Lastly, I'd like to merge flink-language-binding into flink-python.=
 I
> >>>>>
> >>>> can
> >>>
> >>>> go more into detail but the gist of it is that the abstraction just
> >>>>>
> >>>> doesn't
> >>>>
> >>>>> work.
> >>>>>
> >>>>>
> >>>>> On 01.10.2015 12:40, M=C3=A1rton Balassi wrote:
> >>>>>
> >>>>> Great to see streaming graduating. :)
> >>>>>>
> >>>>>> I like the outline, both getting rid of staging, having the exampl=
es
> >>>>>> together and generally flattening the structure are very reasonabl=
e
> to
> >>>>>>
> >>>>> me.
> >>>>
> >>>>> You have listed flink-streaming-examples under
> >>>>>>
> >>>>> flink-streaming-connectors
> >>>>
> >>>>> and left out some less prominent maven modules, but I assume the
> first
> >>>>>>
> >>>>> is
> >>>>
> >>>>> accidental while the second is intentional to make the list a bit
> >>>>>>
> >>>>> briefer.
> >>>>
> >>>>> Best,
> >>>>>>
> >>>>>> Marton
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org>
> >>>>>>
> >>>>> wrote:
> >>>
> >>>> Hi all!
> >>>>>>
> >>>>>>> We are making good headway with reworking the last parts of the
> >>>>>>>
> >>>>>> Window
> >>>
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled out of
> >>>>>>>
> >>>>>> staging.
> >>>>
> >>>>> Since we are reorganizing the projects as part of that, I would
> >>>>>>>
> >>>>>> shift a
> >>>
> >>>> bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-stagin=
g"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not help muc=
h
> to
> >>>>>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in staging
> >>>>>>>
> >>>>>> forever
> >>>>
> >>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>>     -> flink-shaded-hadoop
> >>>>>>>     -> flink-shaded-hadoop2
> >>>>>>>     -> flink-shaded-include-yarn-tests
> >>>>>>>     -> flink-shaded-curator
> >>>>>>>
> >>>>>>> flink-examples
> >>>>>>>     -> (have all examples, Scala and Java, Batch and Streaming)
> >>>>>>>
> >>>>>>> flink-batch-connectors
> >>>>>>>     -> flink-avro
> >>>>>>>     -> flink-jdbc
> >>>>>>>     -> flink-hadoop-compatibility
> >>>>>>>     -> flink-hbase
> >>>>>>>     -> flink-hcatalog
> >>>>>>>
> >>>>>>> flink-streaming-connectors
> >>>>>>>     -> flink-connector-twitter
> >>>>>>>     -> flink-streaming-examples
> >>>>>>>     -> flink-connector-flume
> >>>>>>>     -> flink-connector-kafka
> >>>>>>>     -> flink-connector-elasticsearch
> >>>>>>>     -> flink-connector-rabbitmq
> >>>>>>>     -> flink-connector-filesystem
> >>>>>>>
> >>>>>>> flink-libraries
> >>>>>>>     -> flink-gelly
> >>>>>>>     -> flink-gelly-scala
> >>>>>>>     -> flink-ml
> >>>>>>>     -> flink-table
> >>>>>>>     -> flink-language-binding
> >>>>>>>     -> flink-python
> >>>>>>>
> >>>>>>>
> >>>>>>> flink-scala-shell
> >>>>>>>
> >>>>>>> flink-test-utils
> >>>>>>> flink-tests
> >>>>>>> flink-fs-tests
> >>>>>>>
> >>>>>>> flink-contrib
> >>>>>>>     -> flink-storm-compatibility
> >>>>>>>     -> flink-storm-compatibility-examples
> >>>>>>>     -> flink-streaming-utils
> >>>>>>>     -> flink-tweet-inputformat
> >>>>>>>     -> flink-operator-stats
> >>>>>>>     -> flink-tez
> >>>>>>>
> >>>>>>> flink-quickstart
> >>>>>>>     -> flink-quickstart-java
> >>>>>>>     -> flink-quickstart-scala
> >>>>>>>     -> flink-tez-quickstart
> >>>>>>>
> >>>>>>> flink-yarn
> >>>>>>> flink-yarn-tests
> >>>>>>>
> >>>>>>> flink-dist
> >>>>>>>
> >>>>>>> flink-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>> Let me know if that makes sense!
> >>>>>>>
> >>>>>>> Greetings,
> >>>>>>> Stephan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >
>

--001a1135f0b283ee0305210c2480--