ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Николай Ижиков <nizhikov....@gmail.com>
Subject Integration of Spark and Ignite. Prototype.
Date Mon, 25 Sep 2017 18:14:37 GMT
Hello, guys.

Currently, I’m working on integration between Spark and Ignite [1].

For now, I implement following:
    * Ignite DataSource implementation(IgniteRelationProvider)
    * DataFrame support for Ignite SQL table.
    * IgniteCatalog implementation for a transparent resolving of ignites
SQL tables.

Implementation of it can be found in PR [2]
It would be great if someone provides feedback for a prototype.

I made some examples in PR so you can see how API suppose to be used [3].
[4].

I need some advice. Can you help me?

1. How should this PR be tested?

Of course, I need to provide some unit tests. But what about scalability
tests, etc.
Maybe we need some Yardstick benchmark or similar?
What are your thoughts?
Which scenarios should I consider in the first place?

2. Should we provide Spark Catalog implementation inside Ignite codebase?

A current implementation of Spark Catalog based on *internal Spark API*.
Spark community seems not interested in making Catalog API public or
including Ignite Catalog in Spark code base [5], [6].

*Should we include Spark internal API implementation inside Ignite code
base?*

Or should we consider to include Catalog implementation in some external
module?
That will be created and released outside Ignite?(we still can support and
develop it inside Ignite community).

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://github.com/apache/ignite/pull/2742
[3] https://github.com/apache/ignite/pull/2742/files#diff-
f4ff509cef3018e221394474775e0905
[4] https://github.com/apache/ignite/pull/2742/files#diff-
f2b670497d81e780dfd5098c5dd8a89c
[5] http://apache-spark-developers-list.1001551.n3.
nabble.com/Spark-Core-Custom-Catalog-Integration-between-
Apache-Ignite-and-Apache-Spark-td22452.html
[6] https://issues.apache.org/jira/browse/SPARK-17767

--
Nikolay Izhikov
NIzhikov.dev@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message