ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Pavlukhin <vololo...@gmail.com>
Subject Re: The Spark 2.4 support
Date Mon, 30 Sep 2019 17:17:00 GMT
Hi Alexey,

As an external watcher very far from Ignite Spark integration I would
like to ask a humble question for my understanding. Why this
integration uses Spark internals? Is it a common approach for
integrating with Spark?

пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev <zaleslaw.sin@gmail.com>:
> Hi, Igniters
> I've started the work on the Spark 2.4 support
> We started the discussion here, in
> https://issues.apache.org/jira/browse/IGNITE-12054
> The Spark internals were totally refactored between 2.3 and 2.4 versions,
> main changes touches
>    - External catalog and listeners refactoring
>    - Changes of HAVING operator semantic support
>    - Push-down NULL filters generation in JOIN plans
>    - minor changes in Plan Generation that should be adopted in our
>    integration module
> I propose the initial solution here via creation of new module spark-2.4
> here https://issues.apache.org/jira/browse/IGNITE-12247 and addition of new
> profile spark-2.4 (to avoid possible clashes with another spark versions)
> Also I've transformed ticket to an Umbrella ticket and created a few
> tickets for muted tests (around 7 from 211 tests are muted now)
> Please, if somebody interested in it, make an initial review of modular
> ignite structure and changes (without deep diving into Spark code).
> And yes, the proposed code is a copy-paste of spark-ignite module with a
> few fixes

Best regards,
Ivan Pavlukhin

View raw message