ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Zinoviev <zaleslaw....@gmail.com>
Subject The Spark 2.4 support
Date Mon, 30 Sep 2019 13:06:01 GMT
Hi, Igniters
I've started the work on the Spark 2.4 support

We started the discussion here, in
https://issues.apache.org/jira/browse/IGNITE-12054

The Spark internals were totally refactored between 2.3 and 2.4 versions,
main changes touches

   - External catalog and listeners refactoring
   - Changes of HAVING operator semantic support
   - Push-down NULL filters generation in JOIN plans
   - minor changes in Plan Generation that should be adopted in our
   integration module

I propose the initial solution here via creation of new module spark-2.4
here https://issues.apache.org/jira/browse/IGNITE-12247 and addition of new
profile spark-2.4 (to avoid possible clashes with another spark versions)

Also I've transformed ticket to an Umbrella ticket and created a few
tickets for muted tests (around 7 from 211 tests are muted now)

Please, if somebody interested in it, make an initial review of modular
ignite structure and changes (without deep diving into Spark code).

And yes, the proposed code is a copy-paste of spark-ignite module with a
few fixes

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message