flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Tzoumas <ktzou...@apache.org>
Subject Re: Pulling Streaming out of staging and project restructure
Date Thu, 01 Oct 2015 12:27:01 GMT
+1

I wanted to suggest that we rename modules to fully accept streaming as
first class, qualifying also "batch" as "batch" (e.g., flink-java -->
flink-dataset-java, flink-streaming --> flink-datastream, etc).

This would break maven dependencies (temporary hell :-) so it's not a
decision to take lightly. I'm not strongly advocating for it.


On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.org>
wrote:

> I like it in general. But while we're at it, what is the purpose of the
> flink-tests project, or rather which tests belong there instead of the
> individual projects?
>
> Where would new projects reside in, that previously would have been put
> into flink-staging?
>
> Lastly, I'd like to merge flink-language-binding into flink-python. I can
> go more into detail but the gist of it is that the abstraction just doesn't
> work.
>
>
> On 01.10.2015 12:40, Márton Balassi wrote:
>
>> Great to see streaming graduating. :)
>>
>> I like the outline, both getting rid of staging, having the examples
>> together and generally flattening the structure are very reasonable to me.
>>
>> You have listed flink-streaming-examples under flink-streaming-connectors
>> and left out some less prominent maven modules, but I assume the first is
>> accidental while the second is intentional to make the list a bit briefer.
>>
>> Best,
>>
>> Marton
>>
>>
>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org> wrote:
>>
>> Hi all!
>>>
>>> We are making good headway with reworking the last parts of the Window
>>> API.
>>> After that, the streaming API should be good to be pulled out of staging.
>>>
>>> Since we are reorganizing the projects as part of that, I would shift a
>>> bit
>>> more to bring things a bit more up to date.
>>>
>>> In this restructure, I would like to get rid of the "flink-staging"
>>> project. Anyone who only uses the maven artifacts sees no difference
>>> whether a project is in "staging" or not, so it does not help much to
>>> have
>>> that directory structure.
>>> On the other hand, projects have a tendency to linger in staging forever
>>> (like avro, spargel, hbase, jdbc, ...)
>>>
>>> The new structure could be
>>>
>>> flink-core
>>> flink-java
>>> flink-scala
>>> flink-streaming-core
>>> flink-streaming-scala
>>>
>>> flink-runtime
>>> flink-runtime-web
>>> flink-optimizer
>>> flink-clients
>>>
>>> flink-shaded
>>>    -> flink-shaded-hadoop
>>>    -> flink-shaded-hadoop2
>>>    -> flink-shaded-include-yarn-tests
>>>    -> flink-shaded-curator
>>>
>>> flink-examples
>>>    -> (have all examples, Scala and Java, Batch and Streaming)
>>>
>>> flink-batch-connectors
>>>    -> flink-avro
>>>    -> flink-jdbc
>>>    -> flink-hadoop-compatibility
>>>    -> flink-hbase
>>>    -> flink-hcatalog
>>>
>>> flink-streaming-connectors
>>>    -> flink-connector-twitter
>>>    -> flink-streaming-examples
>>>    -> flink-connector-flume
>>>    -> flink-connector-kafka
>>>    -> flink-connector-elasticsearch
>>>    -> flink-connector-rabbitmq
>>>    -> flink-connector-filesystem
>>>
>>> flink-libraries
>>>    -> flink-gelly
>>>    -> flink-gelly-scala
>>>    -> flink-ml
>>>    -> flink-table
>>>    -> flink-language-binding
>>>    -> flink-python
>>>
>>>
>>> flink-scala-shell
>>>
>>> flink-test-utils
>>> flink-tests
>>> flink-fs-tests
>>>
>>> flink-contrib
>>>    -> flink-storm-compatibility
>>>    -> flink-storm-compatibility-examples
>>>    -> flink-streaming-utils
>>>    -> flink-tweet-inputformat
>>>    -> flink-operator-stats
>>>    -> flink-tez
>>>
>>> flink-quickstart
>>>    -> flink-quickstart-java
>>>    -> flink-quickstart-scala
>>>    -> flink-tez-quickstart
>>>
>>> flink-yarn
>>> flink-yarn-tests
>>>
>>> flink-dist
>>>
>>> flink-benchmark
>>>
>>>
>>> Let me know if that makes sense!
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message