ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Miklosovic (Jira)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-9357) Spark Structured Streaming with Ignite as data source and sink
Date Thu, 21 Nov 2019 10:25:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979158#comment-16979158

Stefan Miklosovic commented on IGNITE-9357:

Hi [~zaleslaw] and [~kukushal], what is the state of DStream implementation of Spark streaming?
I am not mistaken, we currently have:


1) From Kafka to Ignite via its native Java streaming

2) This "structured streaming" which is about Spark SQL stuff

3) Spark integration where you have IgniteRDD abstraction so your e.g. Ignite caches are transparently
viewed as Spark RDDs (or vice versa)


Now, there is not any DStream implementation where e.g I would use Spark Kafka streaming library
and Ignite would be used like sink when I have a stream of messages from e.g. Kafka (but should
be really source-agnostic). There is not also solution where I have Ignite as a source either.


Is Apache Ignite community implementing Ignite on DStreams as we speak?




> Spark Structured Streaming with Ignite as data source and sink
> --------------------------------------------------------------
>                 Key: IGNITE-9357
>                 URL: https://issues.apache.org/jira/browse/IGNITE-9357
>             Project: Ignite
>          Issue Type: New Feature
>          Components: spark
>    Affects Versions: 3.0
>            Reporter: Alexey Kukushkin
>            Assignee: Alexey Zinoviev
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
> We are working on a PoC where we want to use Ignite as a data storage and Spark as a
computation engine. We found that Ignite is supported neither as a source nor as a Sink when
using Spark Structured Streaming, which is a must for us.
> We are enhancing Ignite to support Spark streaming with Ignite. We will send docs and
code for review for the Ignite Community to consider if the community wants to accept this

This message was sent by Atlassian Jira

View raw message