flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raul Kripalani <ra...@apache.org>
Subject Re: Apache Flink <=> Apache Ignite integration
Date Tue, 05 Apr 2016 00:24:32 GMT
On Mon, Apr 4, 2016 at 3:54 PM, Stephan Ewen <sewen@apache.org> wrote:

>   - Sounds like the having Ignite for snapshots should work pretty much
> out
> of the box (via the IGFS)
>   - The source and sink connector sounds like the next logical step. Does
> Ignite have a notion of stream partitions and offsets, to build a
> consistent replay around? This should probably have its dedicated issue and
> discussion thread.
>   - For Ignite as an execution backend - I am not sure how relevant and
> feasible that is. Many DataStream API features make use of the specific
> Flink runtime. For streaming, the runtime is not as decoupled as for
> batch.
>   - I think the parameter server integration would not be part of the
> Flink
> codebase - this is a pretty application specific thing that should be its
> own project and it is actually not tightly coupled to Flink.

Danke, Stephan! I think I'll start with the sink/source connector – reusing
what's already been committed to our codebase.

With regards to source replayability, I plan to integrate Ignite Continuous
Queries as a source. If the user's data objects contain an indexed
ascending numeric or datetime field, we could use such a field as a
"position marker" by launching the query with the appropriate WHERE filter
when a replay is demanded.

Do you have similar use cases with existing connectors?


*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
Blog: raul.io
<http://raul.io/?utm_source=email&utm_medium=email&utm_campaign=apache> |
twitter: @raulvk <https://twitter.com/raulvk>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message