Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: 
 <CAGr9p8A3V0HtLSCHBd3xj3KC5bTSPTv-=DQbccy1UGjMc-YG=w@mail.gmail.com>
References: 
 <CANC1h_u6qtEsF1WCcoU1d38JGd+XTAQWmvp9Stx4vfe68BOjBw@mail.gmail.com>
	<CAKADb_No5y1h3b1zeQPsRaKrtHtx+KAR=jYBJAhJGz=RtWP=fQ@mail.gmail.com>
	<560D0EA2.9020002@apache.org>
	<CAGWx-_ufCdiXbCM44ymB9XmH3j42Xz=9uav1GkW3xy6htCX31w@mail.gmail.com>
	<CAGr9p8A3V0HtLSCHBd3xj3KC5bTSPTv-=DQbccy1UGjMc-YG=w@mail.gmail.com>
Date: Thu, 1 Oct 2015 15:19:12 +0200
Message-ID: 
 <CANC1h_vDb3uTPF39XJCpo34ZjBChpsM3R_78JDXmeb5ThMASwg@mail.gmail.com>
Subject: Re: Pulling Streaming out of staging and project restructure
From: Stephan Ewen <sewen@apache.org>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a113a94c09dd7d805210ae437

--001a113a94c09dd7d805210ae437
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

+1 for Robert's comments.

On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Big +1 for graduating streaming out of staging. It is widely used, also i=
n
> production and we are spending a lot of effort into hardening it.
> I also agree with the proposed new maven module structure.
>
> We have to carefully test the reworked structure for the scripts which ar=
e
> generating the hadoop1 and the scala 2.11 poms (they are transformed usin=
g
> a bunch of bash scripts). I can do that once the PR is open.
>
> @Chesnay: I would be fine with including the language binding into python
> > Where would new projects reside in, that previously would have been put
> into flink-staging?
>
> flink-contrib
>
>
> @Kostas: I understand the idea behind your suggested renaming, but thats
> just a name. I don't think its going to influence how people are seeing
> Flink: It doesn't feel like second class when adding "flink-streaming-cor=
e"
> to the dependencies to me.
> Also, the "flink-datastream-scala" module would depend on
> "flink-dataset-scala", which is kind of weird.
>
>
> I'm wondering whether we should remove the "flink-test-utils" module. I
> don't think its really necessary, because we can put the test jars into t=
he
> flink-tests project and include them using the "test-jar" dependency.
>
>
> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzoumas@apache.org>
> wrote:
>
> > +1
> >
> > I wanted to suggest that we rename modules to fully accept streaming as
> > first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> > flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >
> > This would break maven dependencies (temporary hell :-) so it's not a
> > decision to take lightly. I'm not strongly advocating for it.
> >
> >
> > On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.org>
> > wrote:
> >
> > > I like it in general. But while we're at it, what is the purpose of t=
he
> > > flink-tests project, or rather which tests belong there instead of th=
e
> > > individual projects?
> > >
> > > Where would new projects reside in, that previously would have been p=
ut
> > > into flink-staging?
> > >
> > > Lastly, I'd like to merge flink-language-binding into flink-python. I
> can
> > > go more into detail but the gist of it is that the abstraction just
> > doesn't
> > > work.
> > >
> > >
> > > On 01.10.2015 12:40, M=C3=A1rton Balassi wrote:
> > >
> > >> Great to see streaming graduating. :)
> > >>
> > >> I like the outline, both getting rid of staging, having the examples
> > >> together and generally flattening the structure are very reasonable =
to
> > me.
> > >>
> > >> You have listed flink-streaming-examples under
> > flink-streaming-connectors
> > >> and left out some less prominent maven modules, but I assume the fir=
st
> > is
> > >> accidental while the second is intentional to make the list a bit
> > briefer.
> > >>
> > >> Best,
> > >>
> > >> Marton
> > >>
> > >>
> > >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> > >>
> > >> Hi all!
> > >>>
> > >>> We are making good headway with reworking the last parts of the
> Window
> > >>> API.
> > >>> After that, the streaming API should be good to be pulled out of
> > staging.
> > >>>
> > >>> Since we are reorganizing the projects as part of that, I would
> shift a
> > >>> bit
> > >>> more to bring things a bit more up to date.
> > >>>
> > >>> In this restructure, I would like to get rid of the "flink-staging"
> > >>> project. Anyone who only uses the maven artifacts sees no differenc=
e
> > >>> whether a project is in "staging" or not, so it does not help much =
to
> > >>> have
> > >>> that directory structure.
> > >>> On the other hand, projects have a tendency to linger in staging
> > forever
> > >>> (like avro, spargel, hbase, jdbc, ...)
> > >>>
> > >>> The new structure could be
> > >>>
> > >>> flink-core
> > >>> flink-java
> > >>> flink-scala
> > >>> flink-streaming-core
> > >>> flink-streaming-scala
> > >>>
> > >>> flink-runtime
> > >>> flink-runtime-web
> > >>> flink-optimizer
> > >>> flink-clients
> > >>>
> > >>> flink-shaded
> > >>>    -> flink-shaded-hadoop
> > >>>    -> flink-shaded-hadoop2
> > >>>    -> flink-shaded-include-yarn-tests
> > >>>    -> flink-shaded-curator
> > >>>
> > >>> flink-examples
> > >>>    -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>
> > >>> flink-batch-connectors
> > >>>    -> flink-avro
> > >>>    -> flink-jdbc
> > >>>    -> flink-hadoop-compatibility
> > >>>    -> flink-hbase
> > >>>    -> flink-hcatalog
> > >>>
> > >>> flink-streaming-connectors
> > >>>    -> flink-connector-twitter
> > >>>    -> flink-streaming-examples
> > >>>    -> flink-connector-flume
> > >>>    -> flink-connector-kafka
> > >>>    -> flink-connector-elasticsearch
> > >>>    -> flink-connector-rabbitmq
> > >>>    -> flink-connector-filesystem
> > >>>
> > >>> flink-libraries
> > >>>    -> flink-gelly
> > >>>    -> flink-gelly-scala
> > >>>    -> flink-ml
> > >>>    -> flink-table
> > >>>    -> flink-language-binding
> > >>>    -> flink-python
> > >>>
> > >>>
> > >>> flink-scala-shell
> > >>>
> > >>> flink-test-utils
> > >>> flink-tests
> > >>> flink-fs-tests
> > >>>
> > >>> flink-contrib
> > >>>    -> flink-storm-compatibility
> > >>>    -> flink-storm-compatibility-examples
> > >>>    -> flink-streaming-utils
> > >>>    -> flink-tweet-inputformat
> > >>>    -> flink-operator-stats
> > >>>    -> flink-tez
> > >>>
> > >>> flink-quickstart
> > >>>    -> flink-quickstart-java
> > >>>    -> flink-quickstart-scala
> > >>>    -> flink-tez-quickstart
> > >>>
> > >>> flink-yarn
> > >>> flink-yarn-tests
> > >>>
> > >>> flink-dist
> > >>>
> > >>> flink-benchmark
> > >>>
> > >>>
> > >>> Let me know if that makes sense!
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >>>
> > >>>
> > >
> >
>

--001a113a94c09dd7d805210ae437--