flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
Subject Re: Storm compatibility layer for Flink (first beta available)
Date Mon, 06 Apr 2015 13:40:12 GMT
Done. Shall I open a pull request?

-Matthias


On 04/03/2015 09:32 PM, Robert Metzger wrote:
> As far as I understood git rebase [1], cherry-picking all changes in order
> to the current master is exactly equal to "git rebase flink/master".
> The problem is that you have to resolve all conflicts again. But in this
> case the changes to existing code are pretty small, so that might actually
> work conflict-free.
> 
> 
> [1] http://git-scm.com/docs/git-rebase
> 
> On Fri, Apr 3, 2015 at 9:13 PM, Fabian Hueske <fhueske@gmail.com> wrote:
> 
>> Right now, your commits in your working branch are mixed with commits which
>> are already in pushed to the master branch.
>> Merging this branch to the master branch in order to push it to our master
>> might turn out into a complex merging process.
>> Merging becomes far easier for us, if all commits that you add are placed
>> on top of the latest Flink master branch.
>>
>> To setup a working branch, I would start with the latest Flink master
>> branch and cherry-pick your commits from first to last to that branch. If
>> you want to update the Flink master branch, pull the changes to the master
>> branch and rebase your working branch onto the master branch. That way your
>> commit will be placed on top again.
>>
>> 2015-04-03 0:03 GMT+02:00 Paris Carbone <parisc@kth.se>:
>>
>>> That’s pretty nice Matthias, we could use a compositional API in
>> streaming
>>> that many people are familiar with.
>>> I can also help in some parts, I see some issues we already encountered
>>> while creating the samoa adapter (eg. dealing with circles in the
>>> topology). Thanks again for initiating this!
>>>
>>> Paris
>>>
>>>> On 02 Apr 2015, at 23:14, Gyula Fóra <gyfora@apache.org> wrote:
>>>>
>>>> This sounds amazing :) thanks Matthias!
>>>>
>>>> Tomorrow I will spend some time to look through your work and give some
>>>> comments.
>>>>
>>>> Also I would love to help with this effort so once we merge an initial
>>>> prototype let's open some Jiras and I will pick some up :)
>>>>
>>>> Gyula
>>>>
>>>> On Thursday, April 2, 2015, Márton Balassi <balassi.marton@gmail.com>
>>> wrote:
>>>>
>>>>> Hey Mathias,
>>>>>
>>>>> Thanks, this is a really nice contribution. I just scrolled through
>> the
>>>>> code, but I really like it and big thanks for the the tests for the
>>>>> examples.
>>>>>
>>>>> The rebase Fabian suggested would help a lot when merging.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <fhueske@gmail.com
>>>>> <javascript:;>> wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>> this is really cool!I especially like that you can use Storm code
>>> within
>>>>> a
>>>>>> Flink streaming program :-)
>>>>>>
>>>>>> One thing that might be good to do rather soon is to collect all
your
>>>>>> commits and put them on top of a fresh forked Flink master branch.
>>>>>> When merging we cannot change the history and try to put only
>>>>> fast-forward
>>>>>> commits on top of the master branch.
>>>>>> As time goes on it becomes more likely that you run into merge issues
>>>>> when
>>>>>> cherry-picking the commits.
>>>>>>
>>>>>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <rmetzger@apache.org
>>>>> <javascript:;>>:
>>>>>>
>>>>>>> Hey Henry,
>>>>>>>
>>>>>>> you can check out the files here:
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
>>>>>>> ... so yes, they are located in the flink-streaming directory
..
>> which
>>>>>> is a
>>>>>>> good place for now.
>>>>>>> Once we move flink-streaming out of staging, we might want to
keep
>> the
>>>>>>> storm compat in staging.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
>>> henry.saputra@gmail.com
>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> HI Matthias,
>>>>>>>>
>>>>>>>> Where do you put the code for the Storm compatibility? Under
>> streams
>>>>>>>> module directory?
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
>>>>>>>> <mjsax@informatik.hu-berlin.de <javascript:;>>
wrote:
>>>>>>>>> Hi @all,
>>>>>>>>>
>>>>>>>>> I started to work on an compatibility layer to run Storm
>> Topologies
>>>>>> on
>>>>>>>>> Flink. I just pushed a first beta:
>>>>>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
>>>>>>>>>
>>>>>>>>> Please check it out, and let me know how you like it.
In this
>> first
>>>>>>>>> version, I tried to code without changing too many things
in the
>>>>>>>>> existing code. However, a deeper integration would be
nice.
>>>>>>>>>
>>>>>>>>> There are many things that do not work yet. Currently,
only
>> shuffle
>>>>>> and
>>>>>>>>> fieldsGrouping is supported (and only Storm's default
output
>>>>> stream).
>>>>>>>>> Furthermore, topologies must be "simple", ie, they cannot
be
>>>>>> configured
>>>>>>>>> with a Config object and Storm meta information (ie,
>>>>> TopologyContext)
>>>>>>> is
>>>>>>>>> also not there.
>>>>>>>>>
>>>>>>>>> The layer can be used to integrate existing Spouts and/or
Bolts as
>>>>>>> Flink
>>>>>>>>> operators into a regular Flink program. Furthermore,
a whole Storm
>>>>>>>>> topology can be executed by switching from TopologyBuilder
to
>>>>>>>>> FlinkTopologyBuilder and from
>>>>>> LocalCluster/NimbusClient/StormSubmitter
>>>>>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
>>>>>>>>>
>>>>>>>>> Examples of both cases are given as ITCases.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>  Matthias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
> 


Mime
View raw message