flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Pulling Streaming out of staging and project restructure
Date Thu, 01 Oct 2015 13:28:29 GMT
If we remove flink-staging because projects tend to get stuck there, 
what mechanism prevents the same happening with flink-contrib?

On 01.10.2015 15:19, Stephan Ewen wrote:
> +1 for Robert's comments.
>
> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetzger@apache.org> wrote:
>
>> Big +1 for graduating streaming out of staging. It is widely used, also in
>> production and we are spending a lot of effort into hardening it.
>> I also agree with the proposed new maven module structure.
>>
>> We have to carefully test the reworked structure for the scripts which are
>> generating the hadoop1 and the scala 2.11 poms (they are transformed using
>> a bunch of bash scripts). I can do that once the PR is open.
>>
>> @Chesnay: I would be fine with including the language binding into python
>>> Where would new projects reside in, that previously would have been put
>> into flink-staging?
>>
>> flink-contrib
>>
>>
>> @Kostas: I understand the idea behind your suggested renaming, but thats
>> just a name. I don't think its going to influence how people are seeing
>> Flink: It doesn't feel like second class when adding "flink-streaming-core"
>> to the dependencies to me.
>> Also, the "flink-datastream-scala" module would depend on
>> "flink-dataset-scala", which is kind of weird.
>>
>>
>> I'm wondering whether we should remove the "flink-test-utils" module. I
>> don't think its really necessary, because we can put the test jars into the
>> flink-tests project and include them using the "test-jar" dependency.
>>
>>
>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzoumas@apache.org>
>> wrote:
>>
>>> +1
>>>
>>> I wanted to suggest that we rename modules to fully accept streaming as
>>> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
>>>
>>> This would break maven dependencies (temporary hell :-) so it's not a
>>> decision to take lightly. I'm not strongly advocating for it.
>>>
>>>
>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <chesnay@apache.org>
>>> wrote:
>>>
>>>> I like it in general. But while we're at it, what is the purpose of the
>>>> flink-tests project, or rather which tests belong there instead of the
>>>> individual projects?
>>>>
>>>> Where would new projects reside in, that previously would have been put
>>>> into flink-staging?
>>>>
>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I
>> can
>>>> go more into detail but the gist of it is that the abstraction just
>>> doesn't
>>>> work.
>>>>
>>>>
>>>> On 01.10.2015 12:40, Márton Balassi wrote:
>>>>
>>>>> Great to see streaming graduating. :)
>>>>>
>>>>> I like the outline, both getting rid of staging, having the examples
>>>>> together and generally flattening the structure are very reasonable to
>>> me.
>>>>> You have listed flink-streaming-examples under
>>> flink-streaming-connectors
>>>>> and left out some less prominent maven modules, but I assume the first
>>> is
>>>>> accidental while the second is intentional to make the list a bit
>>> briefer.
>>>>> Best,
>>>>>
>>>>> Marton
>>>>>
>>>>>
>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <sewen@apache.org>
>> wrote:
>>>>> Hi all!
>>>>>> We are making good headway with reworking the last parts of the
>> Window
>>>>>> API.
>>>>>> After that, the streaming API should be good to be pulled out of
>>> staging.
>>>>>> Since we are reorganizing the projects as part of that, I would
>> shift a
>>>>>> bit
>>>>>> more to bring things a bit more up to date.
>>>>>>
>>>>>> In this restructure, I would like to get rid of the "flink-staging"
>>>>>> project. Anyone who only uses the maven artifacts sees no difference
>>>>>> whether a project is in "staging" or not, so it does not help much
to
>>>>>> have
>>>>>> that directory structure.
>>>>>> On the other hand, projects have a tendency to linger in staging
>>> forever
>>>>>> (like avro, spargel, hbase, jdbc, ...)
>>>>>>
>>>>>> The new structure could be
>>>>>>
>>>>>> flink-core
>>>>>> flink-java
>>>>>> flink-scala
>>>>>> flink-streaming-core
>>>>>> flink-streaming-scala
>>>>>>
>>>>>> flink-runtime
>>>>>> flink-runtime-web
>>>>>> flink-optimizer
>>>>>> flink-clients
>>>>>>
>>>>>> flink-shaded
>>>>>>     -> flink-shaded-hadoop
>>>>>>     -> flink-shaded-hadoop2
>>>>>>     -> flink-shaded-include-yarn-tests
>>>>>>     -> flink-shaded-curator
>>>>>>
>>>>>> flink-examples
>>>>>>     -> (have all examples, Scala and Java, Batch and Streaming)
>>>>>>
>>>>>> flink-batch-connectors
>>>>>>     -> flink-avro
>>>>>>     -> flink-jdbc
>>>>>>     -> flink-hadoop-compatibility
>>>>>>     -> flink-hbase
>>>>>>     -> flink-hcatalog
>>>>>>
>>>>>> flink-streaming-connectors
>>>>>>     -> flink-connector-twitter
>>>>>>     -> flink-streaming-examples
>>>>>>     -> flink-connector-flume
>>>>>>     -> flink-connector-kafka
>>>>>>     -> flink-connector-elasticsearch
>>>>>>     -> flink-connector-rabbitmq
>>>>>>     -> flink-connector-filesystem
>>>>>>
>>>>>> flink-libraries
>>>>>>     -> flink-gelly
>>>>>>     -> flink-gelly-scala
>>>>>>     -> flink-ml
>>>>>>     -> flink-table
>>>>>>     -> flink-language-binding
>>>>>>     -> flink-python
>>>>>>
>>>>>>
>>>>>> flink-scala-shell
>>>>>>
>>>>>> flink-test-utils
>>>>>> flink-tests
>>>>>> flink-fs-tests
>>>>>>
>>>>>> flink-contrib
>>>>>>     -> flink-storm-compatibility
>>>>>>     -> flink-storm-compatibility-examples
>>>>>>     -> flink-streaming-utils
>>>>>>     -> flink-tweet-inputformat
>>>>>>     -> flink-operator-stats
>>>>>>     -> flink-tez
>>>>>>
>>>>>> flink-quickstart
>>>>>>     -> flink-quickstart-java
>>>>>>     -> flink-quickstart-scala
>>>>>>     -> flink-tez-quickstart
>>>>>>
>>>>>> flink-yarn
>>>>>> flink-yarn-tests
>>>>>>
>>>>>> flink-dist
>>>>>>
>>>>>> flink-benchmark
>>>>>>
>>>>>>
>>>>>> Let me know if that makes sense!
>>>>>>
>>>>>> Greetings,
>>>>>> Stephan
>>>>>>
>>>>>>


Mime
View raw message