ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Optimization of SQL queries from Spark Data Frame to Ignite
Date Thu, 30 Nov 2017 20:40:24 GMT
Great! Let me know if you need any assistance and/or intermediate review.

-Val

On Thu, Nov 30, 2017 at 12:05 AM, Николай Ижиков <nizhikov.dev@gmail.com>
wrote:

> Valentin,
>
> > Can you please create a separate ticket for the strategy implementation
> then?
>
> Done.
>
> https://issues.apache.org/jira/browse/IGNITE-7077
>
> > Any idea on how long will it take?
>
> I think it will take 2-4 weeks to implement such a strategy.
> I try my best to make a ready to review PR before the end of the year.
>
>
> 30.11.2017 02:13, Valentin Kulichenko пишет:
>
> Nikolay,
>>
>> Can you please create a separate ticket for the strategy implementation
>> then? Any idea on how long will it take?
>>
>> As for querying a partition, both SqlQuery and SqlFieldQuery allow to
>> specify set of partitions to work with (see setPartitions method). I think
>> that should be enough.
>>
>> -Val
>>
>> On Wed, Nov 29, 2017 at 3:39 AM, Vladimir Ozerov <vozerov@gridgain.com>
>> wrote:
>>
>> Hi Nikolay,
>>>
>>> No, it is not possible to get this info from public API, neither we
>>> planned
>>> to expose it. See IGNITE-4509 and commit *fbf0e353* to get better
>>> understanding on how this was implemented.
>>>
>>> Vladimir.
>>>
>>> On Wed, Nov 29, 2017 at 2:01 PM, Николай Ижиков <nizhikov.dev@gmail.com>
>>> wrote:
>>>
>>> Hello, Vladimir.
>>>>
>>>> partition pruning is already implemented in Ignite, so there is no need
>>>>>
>>>> to do this on your own.
>>>>
>>>> Spark work with partitioned data set.
>>>> It is required to provide data partition information to Spark from
>>>> custom
>>>> Data Source(Ignite).
>>>>
>>>> Can I get information about pruned partitions throw some public API?
>>>> Is there a plan or ticket to implement such API?
>>>>
>>>>
>>>>
>>>> 2017-11-29 10:34 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
>>>>
>>>> Nikolay,
>>>>>
>>>>> Regarding p3. - partition pruning is already implemented in Ignite, so
>>>>> there is no need to do this on your own.
>>>>>
>>>>> On Wed, Nov 29, 2017 at 3:23 AM, Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>
>>>>> Nikolay,
>>>>>>
>>>>>> Custom strategy allows to fully process the AST generated by Spark
>>>>>>
>>>>> and
>>>
>>>> convert it to Ignite SQL, so there will be no execution on Spark side
>>>>>>
>>>>> at
>>>>
>>>>> all. This is what we are trying to achieve here. Basically, one will
>>>>>>
>>>>> be
>>>
>>>> able to use DataFrame API to execute queries directly on Ignite. Does
>>>>>>
>>>>> it
>>>>
>>>>> make sense to you?
>>>>>>
>>>>>> I would recommend you to take a look at MemSQL implementation which
>>>>>>
>>>>> does
>>>>
>>>>> similar stuff: https://github.com/memsql/memsql-spark-connector
>>>>>>
>>>>>> Note that this approach will work only if all relations included
in
>>>>>>
>>>>> AST
>>>
>>>> are
>>>>>
>>>>>> Ignite tables. Otherwise, strategy should return null so that Spark
>>>>>>
>>>>> falls
>>>>
>>>>> back to its regular mode. Ignite will be used as regular data source
>>>>>>
>>>>> in
>>>
>>>> this case, and probably it's possible to implement some optimizations
>>>>>>
>>>>> here
>>>>>
>>>>>> as well. However, I never investigated this and it seems like another
>>>>>> separate discussion.
>>>>>>
>>>>>> -Val
>>>>>>
>>>>>> On Tue, Nov 28, 2017 at 9:54 AM, Николай Ижиков <
>>>>>>
>>>>> nizhikov.dev@gmail.com>
>>>>
>>>>> wrote:
>>>>>>
>>>>>> Hello, guys.
>>>>>>>
>>>>>>> I have implemented basic support of Spark Data Frame API [1],
[2]
>>>>>>>
>>>>>> for
>>>
>>>> Ignite.
>>>>>>> Spark provides API for a custom strategy to optimize queries
from
>>>>>>>
>>>>>> spark
>>>>
>>>>> to
>>>>>>
>>>>>>> underlying data source(Ignite).
>>>>>>>
>>>>>>> The goal of optimization(obvious, just to be on the same page):
>>>>>>> Minimize data transfer between Spark and Ignite.
>>>>>>> Speedup query execution.
>>>>>>>
>>>>>>> I see 3 ways to optimize queries:
>>>>>>>
>>>>>>>          1. *Join Reduce* If one make some query that join two
or
>>>>>>>
>>>>>> more
>>>
>>>> Ignite tables, we have to pass all join info to Ignite and transfer
>>>>>>>
>>>>>> to
>>>>
>>>>> Spark only result of table join.
>>>>>>>          To implement it we have to extend current implementation
>>>>>>>
>>>>>> with
>>>
>>>> new
>>>>>
>>>>>> RelationProvider that can generate all kind of joins for two or
>>>>>>>
>>>>>> more
>>>
>>>> tables.
>>>>>>
>>>>>>>          We should add some tests, also.
>>>>>>>          The question is - how join result should be partitioned?
>>>>>>>
>>>>>>>
>>>>>>>          2. *Order by* If one make some query to Ignite table
with
>>>>>>>
>>>>>> order
>>>>
>>>>> by
>>>>>>
>>>>>>> clause we can execute sorting on Ignite side.
>>>>>>>          But it seems that currently Spark doesn’t have any
way to
>>>>>>>
>>>>>> tell
>>>>
>>>>> that partitions already sorted.
>>>>>>>
>>>>>>>
>>>>>>>          3. *Key filter* If one make query with `WHERE key =
XXX` or
>>>>>>>
>>>>>> `WHERE
>>>>>>
>>>>>>> key IN (X, Y, Z)`, we can reduce number of partitions.
>>>>>>>          And query only partitions that store certain key values.
>>>>>>>          Is this kind of optimization already built in Ignite
or I
>>>>>>>
>>>>>> should
>>>>>
>>>>>> implement it by myself?
>>>>>>>
>>>>>>> May be, there is any other way to make queries run faster?
>>>>>>>
>>>>>>> [1] https://spark.apache.org/docs/latest/sql-programming-guide.
>>>>>>>
>>>>>> html
>>>
>>>> [2] https://github.com/apache/ignite/pull/2742
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nikolay Izhikov
>>>> NIzhikov.dev@gmail.com
>>>>
>>>>
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message