Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGt6o8=sAj1Z7m8LbE=xA2y40Yx9_XDOQOmMmhbPR6YjdJMaDA@mail.gmail.com>
References: <23a6488d-733a-51a4-d88e-dbd8a9773513@gmail.com>
 <CABuYRcqh_h09Of_nGTkuO=KMGGDseeVKS6WWMRsJQuzoHeezAQ@mail.gmail.com> <CAGt6o8=sAj1Z7m8LbE=xA2y40Yx9_XDOQOmMmhbPR6YjdJMaDA@mail.gmail.com>
From: =?UTF-8?B?0J3QuNC60L7Qu9Cw0Lkg0JjQttC40LrQvtCy?= <nizhikov.dev@gmail.com>
Date: Wed, 29 Nov 2017 14:01:19 +0300
Message-ID: <CAG7t878owVR1ooMS+DcYJ0_zK3+RkgGm7p0y0+=MG678_MKj3A@mail.gmail.com>
Subject: Re: Optimization of SQL queries from Spark Data Frame to Ignite
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary="94eb2c05eb18305d8b055f1d0e51"
archived-at: Wed, 29 Nov 2017 11:01:27 -0000

--94eb2c05eb18305d8b055f1d0e51
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello, Vladimir.

> partition pruning is already implemented in Ignite, so there is no need
to do this on your own.

Spark work with partitioned data set.
It is required to provide data partition information to Spark from custom
Data Source(Ignite).

Can I get information about pruned partitions throw some public API?
Is there a plan or ticket to implement such API?


2017-11-29 10:34 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:

> Nikolay,
>
> Regarding p3. - partition pruning is already implemented in Ignite, so
> there is no need to do this on your own.
>
> On Wed, Nov 29, 2017 at 3:23 AM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Nikolay,
> >
> > Custom strategy allows to fully process the AST generated by Spark and
> > convert it to Ignite SQL, so there will be no execution on Spark side a=
t
> > all. This is what we are trying to achieve here. Basically, one will be
> > able to use DataFrame API to execute queries directly on Ignite. Does i=
t
> > make sense to you?
> >
> > I would recommend you to take a look at MemSQL implementation which doe=
s
> > similar stuff: https://github.com/memsql/memsql-spark-connector
> >
> > Note that this approach will work only if all relations included in AST
> are
> > Ignite tables. Otherwise, strategy should return null so that Spark fal=
ls
> > back to its regular mode. Ignite will be used as regular data source in
> > this case, and probably it's possible to implement some optimizations
> here
> > as well. However, I never investigated this and it seems like another
> > separate discussion.
> >
> > -Val
> >
> > On Tue, Nov 28, 2017 at 9:54 AM, =D0=9D=D0=B8=D0=BA=D0=BE=D0=BB=D0=B0=
=D0=B9 =D0=98=D0=B6=D0=B8=D0=BA=D0=BE=D0=B2 <nizhikov.dev@gmail.com>
> > wrote:
> >
> > > Hello, guys.
> > >
> > > I have implemented basic support of Spark Data Frame API [1], [2] for
> > > Ignite.
> > > Spark provides API for a custom strategy to optimize queries from spa=
rk
> > to
> > > underlying data source(Ignite).
> > >
> > > The goal of optimization(obvious, just to be on the same page):
> > > Minimize data transfer between Spark and Ignite.
> > > Speedup query execution.
> > >
> > > I see 3 ways to optimize queries:
> > >
> > >         1. *Join Reduce* If one make some query that join two or more
> > > Ignite tables, we have to pass all join info to Ignite and transfer t=
o
> > > Spark only result of table join.
> > >         To implement it we have to extend current implementation with
> new
> > > RelationProvider that can generate all kind of joins for two or more
> > tables.
> > >         We should add some tests, also.
> > >         The question is - how join result should be partitioned?
> > >
> > >
> > >         2. *Order by* If one make some query to Ignite table with ord=
er
> > by
> > > clause we can execute sorting on Ignite side.
> > >         But it seems that currently Spark doesn=E2=80=99t have any wa=
y to tell
> > > that partitions already sorted.
> > >
> > >
> > >         3. *Key filter* If one make query with `WHERE key =3D XXX` or
> > `WHERE
> > > key IN (X, Y, Z)`, we can reduce number of partitions.
> > >         And query only partitions that store certain key values.
> > >         Is this kind of optimization already built in Ignite or I
> should
> > > implement it by myself?
> > >
> > > May be, there is any other way to make queries run faster?
> > >
> > > [1] https://spark.apache.org/docs/latest/sql-programming-guide.html
> > > [2] https://github.com/apache/ignite/pull/2742
> > >
> >
>


--=20
Nikolay Izhikov
NIzhikov.dev@gmail.com

--94eb2c05eb18305d8b055f1d0e51--