flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raul Kripalani <ra...@apache.org>
Subject Apache Flink <=> Apache Ignite integration
Date Wed, 30 Mar 2016 11:48:08 GMT
Hello from the Apache Ignite community!

Last year there was an interesting thread [1] about such integration.
Unfortunately there's been little follow-through, so let's try and fix that
in 2016 ;-)

I'm sure a lot has changed in the Flink community, with the recent
graduation and 1.0 release, so I'd like to make a new (updated) list of
synergies and areas of integration I can think of:

+++ *Ignite as a bidirectional Connector* +++

The first and obvious integration point is Ignite as a source and a sink of
Flink. An Ignite contributor has already sent a pull request [2] to serve
as a sink into Ignite Queues, but I feel this integration can be deeper and
more functional. Moreover, it should be hosted in the Flink source tree as
a Connector (like the Kafka, or ES connectors). Particularly, we could
offer these features:

* As a Flink sink => inject data directly into a cache via a DataStreamer.
* As a Flink source => run a continuous query against one or multiple
caches [4].

+++ *Ignite as a state backend* +++

Either natively [5] or via the IGFS (Ignite Filesystem) interface which can
run as a Hadoop Filesystem [6].

This would allow Flink to store intermediate states in Ignite. I believe
this is what you called "distributed backup for Streaming Operator State"
in the initial exchange, is it?

+++ *Ignite as a DataSet API connector* +++

Ability to use Ignite as a source for batch pipelines, by executing Ignite
SQL queries [7] against a cache and feeding the results into a Flink
pipeline. Basically a batch counterpart to the streaming continuous query
idea above.

+++ *Ignite as an execution backend* +++

You already mentioned this in [1] and I think it makes for a perfect
synergy between both projects, through Ignite's Compute API.

Still agree with this? Any changes since last year I should take into
account?

+++ *Ignite as a parameter server* +++

This was in the initial proposal [1], but it's not clear to me. I have
found references to the idea of a Parameter Server in Flink, but only as
proposed ideas. Was this feature finally implemented, or is it in the
future roadmap?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is just a newer, updated proposal from my side, but I'm sure that both
communities can, and will want to, chime in!

Cheers,

[1]
https://mail-archives.apache.org/mod_mbox/flink-dev/201504.mbox/%3CCANC1h_u__KgsdOo2SZ4M=8jf3zOMozS3XbekQ0erJj9p4WF1zg@mail.gmail.com%3E
[2] https://issues.apache.org/jira/browse/IGNITE-813
[3] https://ignite.apache.org/features/streaming.html
[4] http://apacheignite.gridgain.org/v1.5/docs/continuous-queries
[5] https://apacheignite-fs.readme.io/docs/igfs
[6] https://apacheignite-fs.readme.io/docs/file-system
[7] https://apacheignite.readme.io/docs/sql-queries

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message