ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Integration of Spark and Ignite. Prototype.
Date Tue, 26 Sep 2017 23:20:20 GMT
I will review in the next few days.

-Val

On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <dmagda@apache.org> wrote:

> Hello Nikolay,
>
> This is good news. Finally this capability is coming to Ignite.
>
> Val, Vladimir, could you do a preliminary review?
>
> Answering on your questions.
>
> 1. Yardstick should be enough for performance measurements. As a Spark
> user, I will be curious to know what’s the point of this integration.
> Probably we need to compare Spark + Ignite and Spark + Hive or Spark +
> RDBMS cases.
>
> 2. If Spark community is reluctant let’s include the module in
> ignite-spark integration.
>
> —
> Denis
>
> > On Sep 25, 2017, at 11:14 AM, Николай Ижиков <nizhikov.dev@gmail.com>
> wrote:
> >
> > Hello, guys.
> >
> > Currently, I’m working on integration between Spark and Ignite [1].
> >
> > For now, I implement following:
> >    * Ignite DataSource implementation(IgniteRelationProvider)
> >    * DataFrame support for Ignite SQL table.
> >    * IgniteCatalog implementation for a transparent resolving of ignites
> > SQL tables.
> >
> > Implementation of it can be found in PR [2]
> > It would be great if someone provides feedback for a prototype.
> >
> > I made some examples in PR so you can see how API suppose to be used [3].
> > [4].
> >
> > I need some advice. Can you help me?
> >
> > 1. How should this PR be tested?
> >
> > Of course, I need to provide some unit tests. But what about scalability
> > tests, etc.
> > Maybe we need some Yardstick benchmark or similar?
> > What are your thoughts?
> > Which scenarios should I consider in the first place?
> >
> > 2. Should we provide Spark Catalog implementation inside Ignite codebase?
> >
> > A current implementation of Spark Catalog based on *internal Spark API*.
> > Spark community seems not interested in making Catalog API public or
> > including Ignite Catalog in Spark code base [5], [6].
> >
> > *Should we include Spark internal API implementation inside Ignite code
> > base?*
> >
> > Or should we consider to include Catalog implementation in some external
> > module?
> > That will be created and released outside Ignite?(we still can support
> and
> > develop it inside Ignite community).
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-3084
> > [2] https://github.com/apache/ignite/pull/2742
> > [3] https://github.com/apache/ignite/pull/2742/files#diff-
> > f4ff509cef3018e221394474775e0905
> > [4] https://github.com/apache/ignite/pull/2742/files#diff-
> > f2b670497d81e780dfd5098c5dd8a89c
> > [5] http://apache-spark-developers-list.1001551.n3.
> > nabble.com/Spark-Core-Custom-Catalog-Integration-between-
> > Apache-Ignite-and-Apache-Spark-td22452.html
> > [6] https://issues.apache.org/jira/browse/SPARK-17767
> >
> > --
> > Nikolay Izhikov
> > NIzhikov.dev@gmail.com
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message