flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyf...@apache.org>
Subject Re: Storm compatibility layer for Flink (first beta available)
Date Thu, 02 Apr 2015 21:14:22 GMT
This sounds amazing :) thanks Matthias!

Tomorrow I will spend some time to look through your work and give some
comments.

Also I would love to help with this effort so once we merge an initial
prototype let's open some Jiras and I will pick some up :)

Gyula

On Thursday, April 2, 2015, Márton Balassi <balassi.marton@gmail.com> wrote:

> Hey Mathias,
>
> Thanks, this is a really nice contribution. I just scrolled through the
> code, but I really like it and big thanks for the the tests for the
> examples.
>
> The rebase Fabian suggested would help a lot when merging.
>
>
>
> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <fhueske@gmail.com
> <javascript:;>> wrote:
>
> > Hi Matthias,
> >
> > this is really cool!I especially like that you can use Storm code within
> a
> > Flink streaming program :-)
> >
> > One thing that might be good to do rather soon is to collect all your
> > commits and put them on top of a fresh forked Flink master branch.
> > When merging we cannot change the history and try to put only
> fast-forward
> > commits on top of the master branch.
> > As time goes on it becomes more likely that you run into merge issues
> when
> > cherry-picking the commits.
> >
> > 2015-04-02 21:09 GMT+02:00 Robert Metzger <rmetzger@apache.org
> <javascript:;>>:
> >
> > > Hey Henry,
> > >
> > > you can check out the files here:
> > >
> > >
> >
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> > > ... so yes, they are located in the flink-streaming directory .. which
> > is a
> > > good place for now.
> > > Once we move flink-streaming out of staging, we might want to keep the
> > > storm compat in staging.
> > >
> > >
> > > On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <henry.saputra@gmail.com
> <javascript:;>>
> > > wrote:
> > >
> > > > HI Matthias,
> > > >
> > > > Where do you put the code for the Storm compatibility? Under streams
> > > > module directory?
> > > >
> > > > - Henry
> > > >
> > > > On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> > > > <mjsax@informatik.hu-berlin.de <javascript:;>> wrote:
> > > > > Hi @all,
> > > > >
> > > > > I started to work on an compatibility layer to run Storm Topologies
> > on
> > > > > Flink. I just pushed a first beta:
> > > > > https://github.com/mjsax/flink/tree/flink-storm-compatibility
> > > > >
> > > > > Please check it out, and let me know how you like it. In this first
> > > > > version, I tried to code without changing too many things in the
> > > > > existing code. However, a deeper integration would be nice.
> > > > >
> > > > > There are many things that do not work yet. Currently, only shuffle
> > and
> > > > > fieldsGrouping is supported (and only Storm's default output
> stream).
> > > > > Furthermore, topologies must be "simple", ie, they cannot be
> > configured
> > > > > with a Config object and Storm meta information (ie,
> TopologyContext)
> > > is
> > > > > also not there.
> > > > >
> > > > > The layer can be used to integrate existing Spouts and/or Bolts as
> > > Flink
> > > > > operators into a regular Flink program. Furthermore, a whole Storm
> > > > > topology can be executed by switching from TopologyBuilder to
> > > > > FlinkTopologyBuilder and from
> > LocalCluster/NimbusClient/StormSubmitter
> > > > > to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> > > > >
> > > > > Examples of both cases are given as ITCases.
> > > > >
> > > > >
> > > > > Cheers,
> > > > >   Matthias
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message