flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: Storm compatibility layer for Flink (first beta available)
Date Mon, 06 Apr 2015 15:55:48 GMT
Thanks, Matthias. Yes, please. :)

On Mon, Apr 6, 2015 at 3:40 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:

> Done. Shall I open a pull request?
>
> -Matthias
>
>
> On 04/03/2015 09:32 PM, Robert Metzger wrote:
> > As far as I understood git rebase [1], cherry-picking all changes in
> order
> > to the current master is exactly equal to "git rebase flink/master".
> > The problem is that you have to resolve all conflicts again. But in this
> > case the changes to existing code are pretty small, so that might
> actually
> > work conflict-free.
> >
> >
> > [1] http://git-scm.com/docs/git-rebase
> >
> > On Fri, Apr 3, 2015 at 9:13 PM, Fabian Hueske <fhueske@gmail.com> wrote:
> >
> >> Right now, your commits in your working branch are mixed with commits
> which
> >> are already in pushed to the master branch.
> >> Merging this branch to the master branch in order to push it to our
> master
> >> might turn out into a complex merging process.
> >> Merging becomes far easier for us, if all commits that you add are
> placed
> >> on top of the latest Flink master branch.
> >>
> >> To setup a working branch, I would start with the latest Flink master
> >> branch and cherry-pick your commits from first to last to that branch.
> If
> >> you want to update the Flink master branch, pull the changes to the
> master
> >> branch and rebase your working branch onto the master branch. That way
> your
> >> commit will be placed on top again.
> >>
> >> 2015-04-03 0:03 GMT+02:00 Paris Carbone <parisc@kth.se>:
> >>
> >>> That’s pretty nice Matthias, we could use a compositional API in
> >> streaming
> >>> that many people are familiar with.
> >>> I can also help in some parts, I see some issues we already encountered
> >>> while creating the samoa adapter (eg. dealing with circles in the
> >>> topology). Thanks again for initiating this!
> >>>
> >>> Paris
> >>>
> >>>> On 02 Apr 2015, at 23:14, Gyula Fóra <gyfora@apache.org> wrote:
> >>>>
> >>>> This sounds amazing :) thanks Matthias!
> >>>>
> >>>> Tomorrow I will spend some time to look through your work and give
> some
> >>>> comments.
> >>>>
> >>>> Also I would love to help with this effort so once we merge an initial
> >>>> prototype let's open some Jiras and I will pick some up :)
> >>>>
> >>>> Gyula
> >>>>
> >>>> On Thursday, April 2, 2015, Márton Balassi <balassi.marton@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hey Mathias,
> >>>>>
> >>>>> Thanks, this is a really nice contribution. I just scrolled through
> >> the
> >>>>> code, but I really like it and big thanks for the the tests for
the
> >>>>> examples.
> >>>>>
> >>>>> The rebase Fabian suggested would help a lot when merging.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <fhueske@gmail.com
> >>>>> <javascript:;>> wrote:
> >>>>>
> >>>>>> Hi Matthias,
> >>>>>>
> >>>>>> this is really cool!I especially like that you can use Storm
code
> >>> within
> >>>>> a
> >>>>>> Flink streaming program :-)
> >>>>>>
> >>>>>> One thing that might be good to do rather soon is to collect
all
> your
> >>>>>> commits and put them on top of a fresh forked Flink master branch.
> >>>>>> When merging we cannot change the history and try to put only
> >>>>> fast-forward
> >>>>>> commits on top of the master branch.
> >>>>>> As time goes on it becomes more likely that you run into merge
> issues
> >>>>> when
> >>>>>> cherry-picking the commits.
> >>>>>>
> >>>>>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <rmetzger@apache.org
> >>>>> <javascript:;>>:
> >>>>>>
> >>>>>>> Hey Henry,
> >>>>>>>
> >>>>>>> you can check out the files here:
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> >>>>>>> ... so yes, they are located in the flink-streaming directory
..
> >> which
> >>>>>> is a
> >>>>>>> good place for now.
> >>>>>>> Once we move flink-streaming out of staging, we might want
to keep
> >> the
> >>>>>>> storm compat in staging.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
> >>> henry.saputra@gmail.com
> >>>>> <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> HI Matthias,
> >>>>>>>>
> >>>>>>>> Where do you put the code for the Storm compatibility?
Under
> >> streams
> >>>>>>>> module directory?
> >>>>>>>>
> >>>>>>>> - Henry
> >>>>>>>>
> >>>>>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> >>>>>>>> <mjsax@informatik.hu-berlin.de <javascript:;>>
wrote:
> >>>>>>>>> Hi @all,
> >>>>>>>>>
> >>>>>>>>> I started to work on an compatibility layer to run
Storm
> >> Topologies
> >>>>>> on
> >>>>>>>>> Flink. I just pushed a first beta:
> >>>>>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
> >>>>>>>>>
> >>>>>>>>> Please check it out, and let me know how you like
it. In this
> >> first
> >>>>>>>>> version, I tried to code without changing too many
things in the
> >>>>>>>>> existing code. However, a deeper integration would
be nice.
> >>>>>>>>>
> >>>>>>>>> There are many things that do not work yet. Currently,
only
> >> shuffle
> >>>>>> and
> >>>>>>>>> fieldsGrouping is supported (and only Storm's default
output
> >>>>> stream).
> >>>>>>>>> Furthermore, topologies must be "simple", ie, they
cannot be
> >>>>>> configured
> >>>>>>>>> with a Config object and Storm meta information
(ie,
> >>>>> TopologyContext)
> >>>>>>> is
> >>>>>>>>> also not there.
> >>>>>>>>>
> >>>>>>>>> The layer can be used to integrate existing Spouts
and/or Bolts
> as
> >>>>>>> Flink
> >>>>>>>>> operators into a regular Flink program. Furthermore,
a whole
> Storm
> >>>>>>>>> topology can be executed by switching from TopologyBuilder
to
> >>>>>>>>> FlinkTopologyBuilder and from
> >>>>>> LocalCluster/NimbusClient/StormSubmitter
> >>>>>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> >>>>>>>>>
> >>>>>>>>> Examples of both cases are given as ITCases.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>  Matthias
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message