flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Pulling Streaming out of staging and project restructure
Date Thu, 01 Oct 2015 13:19:12 GMT
+1 for Robert's comments.

On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Big +1 for graduating streaming out of staging. It is widely used, also in
> production and we are spending a lot of effort into hardening it.
> I also agree with the proposed new maven module structure.
>
> We have to carefully test the reworked structure for the scripts which are
> generating the hadoop1 and the scala 2.11 poms (they are transformed using
> a bunch of bash scripts). I can do that once the PR is open.
>
> @Chesnay: I would be fine with including the language binding into python
> > Where would new projects reside in, that previously would have been put
> into flink-staging?
>
> flink-contrib
>
>
> @Kostas: I understand the idea behind your suggested renaming, but thats
> just a name. I don't think its going to influence how people are seeing
> Flink: It doesn't feel like second class when adding "flink-streaming-core"
> to the dependencies to me.
> Also, the "flink-datastream-scala" module would depend on
> "flink-dataset-scala", which is kind of weird.
>
>
> I'm wondering whether we should remove the "flink-test-utils" module. I
> don't think its really necessary, because we can put the test jars into the
> flink-tests project and include them using the "test-jar" dependency.
>
>
> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzoumas@apache.org>
> wrote:
>
> > +1
> >
> > I wanted to suggest that we rename modules to fully accept streaming as
> > first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> > flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >
> > This would break maven dependencies (temporary hell :-) so it's not a
> > decision to take lightly. I'm not strongly advocating for it.
> >
> >
> > On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.org>
> > wrote:
> >
> > > I like it in general. But while we're at it, what is the purpose of the
> > > flink-tests project, or rather which tests belong there instead of the
> > > individual projects?
> > >
> > > Where would new projects reside in, that previously would have been put
> > > into flink-staging?
> > >
> > > Lastly, I'd like to merge flink-language-binding into flink-python. I
> can
> > > go more into detail but the gist of it is that the abstraction just
> > doesn't
> > > work.
> > >
> > >
> > > On 01.10.2015 12:40, Márton Balassi wrote:
> > >
> > >> Great to see streaming graduating. :)
> > >>
> > >> I like the outline, both getting rid of staging, having the examples
> > >> together and generally flattening the structure are very reasonable to
> > me.
> > >>
> > >> You have listed flink-streaming-examples under
> > flink-streaming-connectors
> > >> and left out some less prominent maven modules, but I assume the first
> > is
> > >> accidental while the second is intentional to make the list a bit
> > briefer.
> > >>
> > >> Best,
> > >>
> > >> Marton
> > >>
> > >>
> > >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> > >>
> > >> Hi all!
> > >>>
> > >>> We are making good headway with reworking the last parts of the
> Window
> > >>> API.
> > >>> After that, the streaming API should be good to be pulled out of
> > staging.
> > >>>
> > >>> Since we are reorganizing the projects as part of that, I would
> shift a
> > >>> bit
> > >>> more to bring things a bit more up to date.
> > >>>
> > >>> In this restructure, I would like to get rid of the "flink-staging"
> > >>> project. Anyone who only uses the maven artifacts sees no difference
> > >>> whether a project is in "staging" or not, so it does not help much
to
> > >>> have
> > >>> that directory structure.
> > >>> On the other hand, projects have a tendency to linger in staging
> > forever
> > >>> (like avro, spargel, hbase, jdbc, ...)
> > >>>
> > >>> The new structure could be
> > >>>
> > >>> flink-core
> > >>> flink-java
> > >>> flink-scala
> > >>> flink-streaming-core
> > >>> flink-streaming-scala
> > >>>
> > >>> flink-runtime
> > >>> flink-runtime-web
> > >>> flink-optimizer
> > >>> flink-clients
> > >>>
> > >>> flink-shaded
> > >>>    -> flink-shaded-hadoop
> > >>>    -> flink-shaded-hadoop2
> > >>>    -> flink-shaded-include-yarn-tests
> > >>>    -> flink-shaded-curator
> > >>>
> > >>> flink-examples
> > >>>    -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>
> > >>> flink-batch-connectors
> > >>>    -> flink-avro
> > >>>    -> flink-jdbc
> > >>>    -> flink-hadoop-compatibility
> > >>>    -> flink-hbase
> > >>>    -> flink-hcatalog
> > >>>
> > >>> flink-streaming-connectors
> > >>>    -> flink-connector-twitter
> > >>>    -> flink-streaming-examples
> > >>>    -> flink-connector-flume
> > >>>    -> flink-connector-kafka
> > >>>    -> flink-connector-elasticsearch
> > >>>    -> flink-connector-rabbitmq
> > >>>    -> flink-connector-filesystem
> > >>>
> > >>> flink-libraries
> > >>>    -> flink-gelly
> > >>>    -> flink-gelly-scala
> > >>>    -> flink-ml
> > >>>    -> flink-table
> > >>>    -> flink-language-binding
> > >>>    -> flink-python
> > >>>
> > >>>
> > >>> flink-scala-shell
> > >>>
> > >>> flink-test-utils
> > >>> flink-tests
> > >>> flink-fs-tests
> > >>>
> > >>> flink-contrib
> > >>>    -> flink-storm-compatibility
> > >>>    -> flink-storm-compatibility-examples
> > >>>    -> flink-streaming-utils
> > >>>    -> flink-tweet-inputformat
> > >>>    -> flink-operator-stats
> > >>>    -> flink-tez
> > >>>
> > >>> flink-quickstart
> > >>>    -> flink-quickstart-java
> > >>>    -> flink-quickstart-scala
> > >>>    -> flink-tez-quickstart
> > >>>
> > >>> flink-yarn
> > >>> flink-yarn-tests
> > >>>
> > >>> flink-dist
> > >>>
> > >>> flink-benchmark
> > >>>
> > >>>
> > >>> Let me know if that makes sense!
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >>>
> > >>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message