Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <CAG7t87__uptVJuFxmkmyzsrbADzEEvh9a+=eAWcszjf0ctaQ5A@mail.gmail.com>
References: <CAG7t879kNdXFuvmNC9rmEBpDe88ajp+i7m2CbEb1YEntSXEMqg@mail.gmail.com>
 <E98E53B9-46E3-40D6-A285-406F08431DC7@apache.org> <CABuYRcr5gOjrK=fj0xbf=aTti3Jwe-PTj7iSk41vAssr4zDZ6A@mail.gmail.com>
 <CAG7t87-haLxwLitRhoFQSHmjuj2kNCwtVC2HG3VzqoYWYB5Beg@mail.gmail.com> <CAG7t87__uptVJuFxmkmyzsrbADzEEvh9a+=eAWcszjf0ctaQ5A@mail.gmail.com>
From: Valentin Kulichenko <valentin.kulichenko@gmail.com>
Date: Wed, 11 Oct 2017 20:03:13 -0700
Message-ID: <CABuYRcpBjc3mNhLkUxywxBRU6epQFA+AtL_C=YeESLsGT38qSA@mail.gmail.com>
Subject: Re: Integration of Spark and Ignite. Prototype.
To: =?UTF-8?B?0J3QuNC60L7Qu9Cw0Lkg0JjQttC40LrQvtCy?= <nizhikov.dev@gmail.com>
Cc: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary="001a1141f2a4c862c1055b50c9ee"
archived-at: Thu, 12 Oct 2017 03:10:36 -0000

--001a1141f2a4c862c1055b50c9ee
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Nikolay,

Sorry for delay on this, got a little swamped lately. I will do my best to
review the code this week.

-Val

On Mon, Oct 9, 2017 at 11:48 AM, =D0=9D=D0=B8=D0=BA=D0=BE=D0=BB=D0=B0=D0=B9=
 =D0=98=D0=B6=D0=B8=D0=BA=D0=BE=D0=B2 <nizhikov.dev@gmail.com>
wrote:

> Hello, Valentin.
>
> Did you have a chance to look at my changes?
>
> Now I think I have done almost all required features.
> I want to make some performance test to ensure my implementation work
> properly with a significant amount of data.
> And I definitely need some feedback for my changes.
>
>
> 2017-10-09 18:45 GMT+03:00 =D0=9D=D0=B8=D0=BA=D0=BE=D0=BB=D0=B0=D0=B9 =D0=
=98=D0=B6=D0=B8=D0=BA=D0=BE=D0=B2 <nizhikov.dev@gmail.com>:
>
>> Hello, guys.
>>
>> Which version of Spark do we want to use?
>>
>> 1. Currently, Ignite depends on Spark 2.1.0.
>>
>>     * Can be run on JDK 7.
>>     * Still supported: 2.1.2 will be released soon.
>>
>> 2. Latest Spark version is 2.2.0.
>>
>>     * Can be run only on JDK 8+
>>     * Released Jul 11, 2017.
>>     * Already supported by huge vendors(Amazon for example).
>>
>> Note that in IGNITE-3084 I implement some internal Spark API.
>> So It will take some effort to switch between Spark 2.1 and 2.2
>>
>>
>> 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko <
>> valentin.kulichenko@gmail.com>:
>>
>>> I will review in the next few days.
>>>
>>> -Val
>>>
>>> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <dmagda@apache.org> wrote:
>>>
>>> > Hello Nikolay,
>>> >
>>> > This is good news. Finally this capability is coming to Ignite.
>>> >
>>> > Val, Vladimir, could you do a preliminary review?
>>> >
>>> > Answering on your questions.
>>> >
>>> > 1. Yardstick should be enough for performance measurements. As a Spar=
k
>>> > user, I will be curious to know what=E2=80=99s the point of this inte=
gration.
>>> > Probably we need to compare Spark + Ignite and Spark + Hive or Spark =
+
>>> > RDBMS cases.
>>> >
>>> > 2. If Spark community is reluctant let=E2=80=99s include the module i=
n
>>> > ignite-spark integration.
>>> >
>>> > =E2=80=94
>>> > Denis
>>> >
>>> > > On Sep 25, 2017, at 11:14 AM, =D0=9D=D0=B8=D0=BA=D0=BE=D0=BB=D0=B0=
=D0=B9 =D0=98=D0=B6=D0=B8=D0=BA=D0=BE=D0=B2 <nizhikov.dev@gmail.com
>>> >
>>> > wrote:
>>> > >
>>> > > Hello, guys.
>>> > >
>>> > > Currently, I=E2=80=99m working on integration between Spark and Ign=
ite [1].
>>> > >
>>> > > For now, I implement following:
>>> > >    * Ignite DataSource implementation(IgniteRelationProvider)
>>> > >    * DataFrame support for Ignite SQL table.
>>> > >    * IgniteCatalog implementation for a transparent resolving of
>>> ignites
>>> > > SQL tables.
>>> > >
>>> > > Implementation of it can be found in PR [2]
>>> > > It would be great if someone provides feedback for a prototype.
>>> > >
>>> > > I made some examples in PR so you can see how API suppose to be use=
d
>>> [3].
>>> > > [4].
>>> > >
>>> > > I need some advice. Can you help me?
>>> > >
>>> > > 1. How should this PR be tested?
>>> > >
>>> > > Of course, I need to provide some unit tests. But what about
>>> scalability
>>> > > tests, etc.
>>> > > Maybe we need some Yardstick benchmark or similar?
>>> > > What are your thoughts?
>>> > > Which scenarios should I consider in the first place?
>>> > >
>>> > > 2. Should we provide Spark Catalog implementation inside Ignite
>>> codebase?
>>> > >
>>> > > A current implementation of Spark Catalog based on *internal Spark
>>> API*.
>>> > > Spark community seems not interested in making Catalog API public o=
r
>>> > > including Ignite Catalog in Spark code base [5], [6].
>>> > >
>>> > > *Should we include Spark internal API implementation inside Ignite
>>> code
>>> > > base?*
>>> > >
>>> > > Or should we consider to include Catalog implementation in some
>>> external
>>> > > module?
>>> > > That will be created and released outside Ignite?(we still can
>>> support
>>> > and
>>> > > develop it inside Ignite community).
>>> > >
>>> > > [1] https://issues.apache.org/jira/browse/IGNITE-3084
>>> > > [2] https://github.com/apache/ignite/pull/2742
>>> > > [3] https://github.com/apache/ignite/pull/2742/files#diff-
>>> > > f4ff509cef3018e221394474775e0905
>>> > > [4] https://github.com/apache/ignite/pull/2742/files#diff-
>>> > > f2b670497d81e780dfd5098c5dd8a89c
>>> > > [5] http://apache-spark-developers-list.1001551.n3.
>>> > > nabble.com/Spark-Core-Custom-Catalog-Integration-between-
>>> > > Apache-Ignite-and-Apache-Spark-td22452.html
>>> > > [6] https://issues.apache.org/jira/browse/SPARK-17767
>>> > >
>>> > > --
>>> > > Nikolay Izhikov
>>> > > NIzhikov.dev@gmail.com
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Nikolay Izhikov
>> NIzhikov.dev@gmail.com
>>
>
>
>
> --
> Nikolay Izhikov
> NIzhikov.dev@gmail.com
>

--001a1141f2a4c862c1055b50c9ee--