flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Apache Flink <=> Apache Ignite integration
Date Tue, 05 Apr 2016 08:40:55 GMT
Hi Raul!

Concerning the source connector and position marker: Great idea!
The FlinkKafkaConsumer uses pretty much the same trick - the
offset-per-partition is used to filter during replays.

Greetings,
Stephan



On Tue, Apr 5, 2016 at 2:24 AM, Raul Kripalani <raulk@apache.org> wrote:

> On Mon, Apr 4, 2016 at 3:54 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> >
> >   - Sounds like the having Ignite for snapshots should work pretty much
> > out
> > of the box (via the IGFS)
> >   - The source and sink connector sounds like the next logical step. Does
> > Ignite have a notion of stream partitions and offsets, to build a
> > consistent replay around? This should probably have its dedicated issue
> and
> > discussion thread.
> >
> >   - For Ignite as an execution backend - I am not sure how relevant and
> > feasible that is. Many DataStream API features make use of the specific
> > Flink runtime. For streaming, the runtime is not as decoupled as for
> > batch.
> >   - I think the parameter server integration would not be part of the
> > Flink
> > codebase - this is a pretty application specific thing that should be its
> > own project and it is actually not tightly coupled to Flink.
>
>
> Danke, Stephan! I think I'll start with the sink/source connector – reusing
> what's already been committed to our codebase.
>
> With regards to source replayability, I plan to integrate Ignite Continuous
> Queries as a source. If the user's data objects contain an indexed
> ascending numeric or datetime field, we could use such a field as a
> "position marker" by launching the query with the appropriate WHERE filter
> when a replay is demanded.
>
> Do you have similar use cases with existing connectors?
>
> Cheers,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> Blog: raul.io
> <http://raul.io/?utm_source=email&utm_medium=email&utm_campaign=apache>
|
> twitter: @raulvk <https://twitter.com/raulvk>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message