flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Apache Flink <=> Apache Ignite integration
Date Tue, 19 Jul 2016 08:53:04 GMT
Hi Saikat,

Thanks for this contribution.

Anton V., please review the following contribution.

—
Denis

> On Jul 16, 2016, at 11:09 PM, Saikat Maitra <saikat.maitra@gmail.com> wrote:
> 
> Hi
> 
> I have raised a PR for the following scope.
> 
> As a Flink source => run a continuous query against one or multiple
> caches
> 
> PR https://github.com/apache/ignite/pull/870
> Jira https://issues.apache.org/jira/browse/IGNITE-3303
> 
> Please review and share feedback.
> 
> Regards
> Saikat
> 
> 
> On Mon, Apr 4, 2016 at 8:24 PM, Stephan Ewen <sewen@apache.org> wrote:
> 
>> Hi!
>> 
>>  - Sounds like the having Ignite for snapshots should work pretty much out
>> of the box (via the IGFS)
>>  - The source and sink connector sounds like the next logical step. Does
>> Ignite have a notion of stream partitions and offsets, to build a
>> consistent replay around? This should probably have its dedicated issue and
>> discussion thread.
>> 
>>  - For Ignite as an execution backend - I am not sure how relevant and
>> feasible that is. Many DataStream API features make use of the specific
>> Flink runtime. For streaming, the runtime is not as decoupled as for batch.
>>  - I think the parameter server integration would not be part of the Flink
>> codebase - this is a pretty application specific thing that should be its
>> own project and it is actually not tightly coupled to Flink.
>> 
>> Greetings,
>> Stephan
>> 
>> 
>> On Mon, Apr 4, 2016 at 4:35 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>> 
>>> Hi Raul,
>>> 
>>> thanks a lot for reaching out to the Flink community.
>>> I'm really excited to see a Flink connector in Ignite. If you feel that
>> the
>>> connector would be more suitable for our "connector library" feel free to
>>> open a JIRA and open a pull request.
>>> 
>>> Were there requests in the Ignite community to have an integration with
>>> Flink?
>>> 
>>> 
>>> 
>>> On Thu, Mar 31, 2016 at 5:20 PM, Saikat Maitra <saikat.maitra@gmail.com>
>>> wrote:
>>> 
>>>> Hi ,
>>>> 
>>>> I agree with Roman and Raul.
>>>> https://issues.apache.org/jira/browse/IGNITE-813 allows injecting data
>>> to
>>>> into cache via Data Streamer. Integrating with Ignite FileSystem for
>>> source
>>>> and sink will allow for bidirectional connector. It will also allow
>>> easier
>>>> implementation for DataStream transformations over Ignite FileSystem.
>>>> 
>>>> Regards
>>>> Saikat
>>>> 
>>>> On Thu, Mar 31, 2016 at 2:44 PM, Aljoscha Krettek <aljoscha@apache.org
>>> 
>>>> wrote:
>>>> 
>>>>> Hi,
>>>>> it should already be possible to use the Ignite FileSystem to store
>>> state
>>>>> since we just use the HDFS FileSystem interface for that. Of course,
>>> one
>>>>> would have to properly set up the jars and paths and everything for
>>> Flink
>>>>> to pick up the IGFS classes.
>>>>> 
>>>>> Cheers,
>>>>> Aljoscha
>>>>> 
>>>>> On Wed, 30 Mar 2016 at 16:50 Raul Kripalani <raulk@apache.org>
>> wrote:
>>>>> 
>>>>>> On Wed, Mar 30, 2016 at 2:20 PM, Roman <rshtykh@yahoo.com.invalid>
>>>>> wrote:
>>>>>> 
>>>>>>> Raul,
>>>>>>> 
>>>>>>> Small comment from me.
>>>>>>> 
>>>>>>>> * As a Flink sink => inject data directly into a cache
via a
>>>>>> DataStreamer.
>>>>>>> After reviews, IGNITE-813 is exactly this functionality.
>>>>>>> 
>>>>>>> 
>>>>>> That's cool, Roman! The idea would be to host these (richer)
>> modules
>>> as
>>>>>> Flink connectors, like they do with others:
>>>>>> 
>>>>>> 
>>> https://github.com/apache/flink/tree/master/flink-streaming-connectors
>>>>>> https://github.com/apache/flink/tree/master/flink-batch-connectors
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message