flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Pulling Streaming out of staging and project restructure
Date Thu, 01 Oct 2015 14:48:19 GMT
@Chesnay: Nothing prevents projects from getting stuck there. But at least
we don't have two staging repositories (dist, staging). Also I would not
say that there has been no graduation out of staging. Yarn was also there
once, streaming and gelly are currently leaving it. So I would actually say
it worked.

One more thing: Can we actually put the web submission client into a
separate module until we have something new and better?

It does really not fit into "flink-clients". It makes Flink users pull the
jetty dependencies even if they never use them for example when running
Flink from an IDE.


On Thu, Oct 1, 2015 at 4:45 PM, Kostas Tzoumas <ktzoumas@apache.org> wrote:

> +1 to Robert and practicality :-)
>
> As I said before, I do not feel strongly about this, I was torn myself.
>
> On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
> > If we remove flink-staging because projects tend to get stuck there, what
> > mechanism prevents the same happening with flink-contrib?
> >
> >
> > On 01.10.2015 15:19, Stephan Ewen wrote:
> >
> >> +1 for Robert's comments.
> >>
> >> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetzger@apache.org>
> >> wrote:
> >>
> >> Big +1 for graduating streaming out of staging. It is widely used, also
> in
> >>> production and we are spending a lot of effort into hardening it.
> >>> I also agree with the proposed new maven module structure.
> >>>
> >>> We have to carefully test the reworked structure for the scripts which
> >>> are
> >>> generating the hadoop1 and the scala 2.11 poms (they are transformed
> >>> using
> >>> a bunch of bash scripts). I can do that once the PR is open.
> >>>
> >>> @Chesnay: I would be fine with including the language binding into
> python
> >>>
> >>>> Where would new projects reside in, that previously would have been
> put
> >>>>
> >>> into flink-staging?
> >>>
> >>> flink-contrib
> >>>
> >>>
> >>> @Kostas: I understand the idea behind your suggested renaming, but
> thats
> >>> just a name. I don't think its going to influence how people are seeing
> >>> Flink: It doesn't feel like second class when adding
> >>> "flink-streaming-core"
> >>> to the dependencies to me.
> >>> Also, the "flink-datastream-scala" module would depend on
> >>> "flink-dataset-scala", which is kind of weird.
> >>>
> >>>
> >>> I'm wondering whether we should remove the "flink-test-utils" module. I
> >>> don't think its really necessary, because we can put the test jars into
> >>> the
> >>> flink-tests project and include them using the "test-jar" dependency.
> >>>
> >>>
> >>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzoumas@apache.org>
> >>> wrote:
> >>>
> >>> +1
> >>>>
> >>>> I wanted to suggest that we rename modules to fully accept streaming
> as
> >>>> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> >>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >>>>
> >>>> This would break maven dependencies (temporary hell :-) so it's not
a
> >>>> decision to take lightly. I'm not strongly advocating for it.
> >>>>
> >>>>
> >>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.org
> >
> >>>> wrote:
> >>>>
> >>>> I like it in general. But while we're at it, what is the purpose of
> the
> >>>>> flink-tests project, or rather which tests belong there instead
of
> the
> >>>>> individual projects?
> >>>>>
> >>>>> Where would new projects reside in, that previously would have been
> put
> >>>>> into flink-staging?
> >>>>>
> >>>>> Lastly, I'd like to merge flink-language-binding into flink-python.
I
> >>>>>
> >>>> can
> >>>
> >>>> go more into detail but the gist of it is that the abstraction just
> >>>>>
> >>>> doesn't
> >>>>
> >>>>> work.
> >>>>>
> >>>>>
> >>>>> On 01.10.2015 12:40, Márton Balassi wrote:
> >>>>>
> >>>>> Great to see streaming graduating. :)
> >>>>>>
> >>>>>> I like the outline, both getting rid of staging, having the
examples
> >>>>>> together and generally flattening the structure are very reasonable
> to
> >>>>>>
> >>>>> me.
> >>>>
> >>>>> You have listed flink-streaming-examples under
> >>>>>>
> >>>>> flink-streaming-connectors
> >>>>
> >>>>> and left out some less prominent maven modules, but I assume the
> first
> >>>>>>
> >>>>> is
> >>>>
> >>>>> accidental while the second is intentional to make the list a bit
> >>>>>>
> >>>>> briefer.
> >>>>
> >>>>> Best,
> >>>>>>
> >>>>>> Marton
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org>
> >>>>>>
> >>>>> wrote:
> >>>
> >>>> Hi all!
> >>>>>>
> >>>>>>> We are making good headway with reworking the last parts
of the
> >>>>>>>
> >>>>>> Window
> >>>
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled
out of
> >>>>>>>
> >>>>>> staging.
> >>>>
> >>>>> Since we are reorganizing the projects as part of that, I would
> >>>>>>>
> >>>>>> shift a
> >>>
> >>>> bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-staging"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not
help much
> to
> >>>>>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in
staging
> >>>>>>>
> >>>>>> forever
> >>>>
> >>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>>     -> flink-shaded-hadoop
> >>>>>>>     -> flink-shaded-hadoop2
> >>>>>>>     -> flink-shaded-include-yarn-tests
> >>>>>>>     -> flink-shaded-curator
> >>>>>>>
> >>>>>>> flink-examples
> >>>>>>>     -> (have all examples, Scala and Java, Batch and
Streaming)
> >>>>>>>
> >>>>>>> flink-batch-connectors
> >>>>>>>     -> flink-avro
> >>>>>>>     -> flink-jdbc
> >>>>>>>     -> flink-hadoop-compatibility
> >>>>>>>     -> flink-hbase
> >>>>>>>     -> flink-hcatalog
> >>>>>>>
> >>>>>>> flink-streaming-connectors
> >>>>>>>     -> flink-connector-twitter
> >>>>>>>     -> flink-streaming-examples
> >>>>>>>     -> flink-connector-flume
> >>>>>>>     -> flink-connector-kafka
> >>>>>>>     -> flink-connector-elasticsearch
> >>>>>>>     -> flink-connector-rabbitmq
> >>>>>>>     -> flink-connector-filesystem
> >>>>>>>
> >>>>>>> flink-libraries
> >>>>>>>     -> flink-gelly
> >>>>>>>     -> flink-gelly-scala
> >>>>>>>     -> flink-ml
> >>>>>>>     -> flink-table
> >>>>>>>     -> flink-language-binding
> >>>>>>>     -> flink-python
> >>>>>>>
> >>>>>>>
> >>>>>>> flink-scala-shell
> >>>>>>>
> >>>>>>> flink-test-utils
> >>>>>>> flink-tests
> >>>>>>> flink-fs-tests
> >>>>>>>
> >>>>>>> flink-contrib
> >>>>>>>     -> flink-storm-compatibility
> >>>>>>>     -> flink-storm-compatibility-examples
> >>>>>>>     -> flink-streaming-utils
> >>>>>>>     -> flink-tweet-inputformat
> >>>>>>>     -> flink-operator-stats
> >>>>>>>     -> flink-tez
> >>>>>>>
> >>>>>>> flink-quickstart
> >>>>>>>     -> flink-quickstart-java
> >>>>>>>     -> flink-quickstart-scala
> >>>>>>>     -> flink-tez-quickstart
> >>>>>>>
> >>>>>>> flink-yarn
> >>>>>>> flink-yarn-tests
> >>>>>>>
> >>>>>>> flink-dist
> >>>>>>>
> >>>>>>> flink-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>> Let me know if that makes sense!
> >>>>>>>
> >>>>>>> Greetings,
> >>>>>>> Stephan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message