ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Zinoviev (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (IGNITE-12054) Upgrade Spark module to 2.4
Date Wed, 25 Sep 2019 13:20:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937715#comment-16937715
] 

Aleksey Zinoviev edited comment on IGNITE-12054 at 9/25/19 1:19 PM:
--------------------------------------------------------------------

I've started the R&D about 2.4 version, fixed a few minor changes and posted here a few
thoughts about current problems here, in Jira comments.

 

An ExternalCatalog was refactored here [https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612] and
all listener properties were inherited in ExternalCatalogWithListener.

 

Nobody yet inherited from this class on Github, the known implementations are HiveExternalCatalog
and MemoryExternalCatalog (both of them doesn't support listeners and events)

 

Also, people in Spark ML couldn't solve the same problem

[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3CJIRA.13144856.1520975543000.147283.1544598241661@Atlassian.JIRA%3E]


was (Author: zaleslaw):
I've started the R&D about 2.4 version, fixed a few minor changes and posted here a few
thoughts about current problems here, in Jira comments.

 

An ExternalCatalog was [link title|[https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612]] and
all listener properties were inherited in ExternalCatalogWithListener.

 

Nobody yet inherited from this class on Github, the known implementations are HiveExternalCatalog
and MemoryExternalCatalog (both of them doesn't support listeners and events)

 

Also, people in Spark ML couldn't solve the same problem

[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3CJIRA.13144856.1520975543000.147283.1544598241661@Atlassian.JIRA%3E]

> Upgrade Spark module to 2.4
> ---------------------------
>
>                 Key: IGNITE-12054
>                 URL: https://issues.apache.org/jira/browse/IGNITE-12054
>             Project: Ignite
>          Issue Type: Task
>          Components: spark
>    Affects Versions: 2.7.5
>            Reporter: Denis Magda
>            Assignee: Nikolay Izhikov
>            Priority: Blocker
>             Fix For: 2.8
>
>
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as a separate
module that can support multiple Spark versions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message