ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Николай Ижиков <nizhikov....@gmail.com>
Subject Integration of Spark and Ignite. Prototype.
Date Mon, 25 Sep 2017 18:14:37 GMT
Hello, guys.

Currently, I’m working on integration between Spark and Ignite [1].

For now, I implement following:
    * Ignite DataSource implementation(IgniteRelationProvider)
    * DataFrame support for Ignite SQL table.
    * IgniteCatalog implementation for a transparent resolving of ignites
SQL tables.

Implementation of it can be found in PR [2]
It would be great if someone provides feedback for a prototype.

I made some examples in PR so you can see how API suppose to be used [3].

I need some advice. Can you help me?

1. How should this PR be tested?

Of course, I need to provide some unit tests. But what about scalability
tests, etc.
Maybe we need some Yardstick benchmark or similar?
What are your thoughts?
Which scenarios should I consider in the first place?

2. Should we provide Spark Catalog implementation inside Ignite codebase?

A current implementation of Spark Catalog based on *internal Spark API*.
Spark community seems not interested in making Catalog API public or
including Ignite Catalog in Spark code base [5], [6].

*Should we include Spark internal API implementation inside Ignite code

Or should we consider to include Catalog implementation in some external
That will be created and released outside Ignite?(we still can support and
develop it inside Ignite community).

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://github.com/apache/ignite/pull/2742
[3] https://github.com/apache/ignite/pull/2742/files#diff-
[4] https://github.com/apache/ignite/pull/2742/files#diff-
[5] http://apache-spark-developers-list.1001551.n3.
[6] https://issues.apache.org/jira/browse/SPARK-17767

Nikolay Izhikov

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message