From issues-return-101654-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Nov 21 10:25:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 32BA2180660 for ; Thu, 21 Nov 2019 11:25:02 +0100 (CET) Received: (qmail 68772 invoked by uid 500); 21 Nov 2019 10:25:01 -0000 Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list issues@ignite.apache.org Received: (qmail 68750 invoked by uid 99); 21 Nov 2019 10:25:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2019 10:25:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id DC229E2F4F for ; Thu, 21 Nov 2019 10:25:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 5A34E780479 for ; Thu, 21 Nov 2019 10:25:00 +0000 (UTC) Date: Thu, 21 Nov 2019 10:25:00 +0000 (UTC) From: "Stefan Miklosovic (Jira)" To: issues@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (IGNITE-9357) Spark Structured Streaming with Ignite as data source and sink MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/IGNITE-9357?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D169= 79158#comment-16979158 ]=20 Stefan Miklosovic commented on IGNITE-9357: ------------------------------------------- Hi [~zaleslaw]=C2=A0and [~kukushal], what is the state of DStream implement= ation of Spark streaming? I am not mistaken, we currently have: =C2=A0 1) From Kafka to Ignite via its native Java streaming 2) This "structured streaming" which is about Spark SQL stuff 3) Spark integration where you have IgniteRDD abstraction so your e.g. Igni= te caches are transparently viewed as Spark RDDs (or vice versa) =C2=A0 Now, there is not any DStream implementation where e.g I would use Spark Ka= fka streaming library and Ignite would be used like sink when I have a stre= am of messages from e.g. Kafka (but should be really source-agnostic). Ther= e is not also solution where I have Ignite as a source either. =C2=A0 Is Apache Ignite community implementing Ignite on DStreams as we speak? =C2=A0 Thanks! =C2=A0 > Spark Structured Streaming with Ignite as data source and sink > -------------------------------------------------------------- > > Key: IGNITE-9357 > URL: https://issues.apache.org/jira/browse/IGNITE-9357 > Project: Ignite > Issue Type: New Feature > Components: spark > Affects Versions: 3.0 > Reporter: Alexey Kukushkin > Assignee: Alexey Zinoviev > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We are working on a PoC where we want to use Ignite as a data storage and= Spark as a computation engine. We found that Ignite is supported neither a= s a source nor as a Sink when using Spark Structured Streaming, which is a = must for us. > We are enhancing Ignite to support Spark streaming with Ignite. We will s= end docs and code for review for the Ignite Community to consider if the co= mmunity wants to accept this feature.=C2=A0 -- This message was sent by Atlassian Jira (v8.3.4#803005)