ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: Apache Flink <=> Apache Ignite integration
Date Sat, 16 Jul 2016 20:09:19 GMT
Hi

I have raised a PR for the following scope.

As a Flink source => run a continuous query against one or multiple
caches

PR https://github.com/apache/ignite/pull/870
Jira https://issues.apache.org/jira/browse/IGNITE-3303

Please review and share feedback.

Regards
Saikat


On Mon, Apr 4, 2016 at 8:24 PM, Stephan Ewen <sewen@apache.org> wrote:

> Hi!
>
>   - Sounds like the having Ignite for snapshots should work pretty much out
> of the box (via the IGFS)
>   - The source and sink connector sounds like the next logical step. Does
> Ignite have a notion of stream partitions and offsets, to build a
> consistent replay around? This should probably have its dedicated issue and
> discussion thread.
>
>   - For Ignite as an execution backend - I am not sure how relevant and
> feasible that is. Many DataStream API features make use of the specific
> Flink runtime. For streaming, the runtime is not as decoupled as for batch.
>   - I think the parameter server integration would not be part of the Flink
> codebase - this is a pretty application specific thing that should be its
> own project and it is actually not tightly coupled to Flink.
>
> Greetings,
> Stephan
>
>
> On Mon, Apr 4, 2016 at 4:35 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Hi Raul,
> >
> > thanks a lot for reaching out to the Flink community.
> > I'm really excited to see a Flink connector in Ignite. If you feel that
> the
> > connector would be more suitable for our "connector library" feel free to
> > open a JIRA and open a pull request.
> >
> > Were there requests in the Ignite community to have an integration with
> > Flink?
> >
> >
> >
> > On Thu, Mar 31, 2016 at 5:20 PM, Saikat Maitra <saikat.maitra@gmail.com>
> > wrote:
> >
> > > Hi ,
> > >
> > > I agree with Roman and Raul.
> > > https://issues.apache.org/jira/browse/IGNITE-813 allows injecting data
> > to
> > > into cache via Data Streamer. Integrating with Ignite FileSystem for
> > source
> > > and sink will allow for bidirectional connector. It will also allow
> > easier
> > > implementation for DataStream transformations over Ignite FileSystem.
> > >
> > > Regards
> > > Saikat
> > >
> > > On Thu, Mar 31, 2016 at 2:44 PM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > >
> > > > Hi,
> > > > it should already be possible to use the Ignite FileSystem to store
> > state
> > > > since we just use the HDFS FileSystem interface for that. Of course,
> > one
> > > > would have to properly set up the jars and paths and everything for
> > Flink
> > > > to pick up the IGFS classes.
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Wed, 30 Mar 2016 at 16:50 Raul Kripalani <raulk@apache.org>
> wrote:
> > > >
> > > > > On Wed, Mar 30, 2016 at 2:20 PM, Roman <rshtykh@yahoo.com.invalid>
> > > > wrote:
> > > > >
> > > > > > Raul,
> > > > > >
> > > > > > Small comment from me.
> > > > > >
> > > > > > >* As a Flink sink => inject data directly into a cache
via a
> > > > > DataStreamer.
> > > > > > After reviews, IGNITE-813 is exactly this functionality.
> > > > > >
> > > > > >
> > > > > That's cool, Roman! The idea would be to host these (richer)
> modules
> > as
> > > > > Flink connectors, like they do with others:
> > > > >
> > > > >
> > https://github.com/apache/flink/tree/master/flink-streaming-connectors
> > > > > https://github.com/apache/flink/tree/master/flink-batch-connectors
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message