ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Zinoviev <zaleslaw....@gmail.com>
Subject Re: The Spark 2.4 support
Date Mon, 30 Sep 2019 17:29:11 GMT
Yes, as I understand it uses Spark internals from the first commit)))
The reason - we take Spark SQL query execution plan and try to execute it
on Ignite cluster
Also we inherit a lot of Developer API related classes that could be
unstable. Spark has no good point for extension and this is a reason why we
should go deeper

пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin <vololo100@gmail.com>:

> Hi Alexey,
>
> As an external watcher very far from Ignite Spark integration I would
> like to ask a humble question for my understanding. Why this
> integration uses Spark internals? Is it a common approach for
> integrating with Spark?
>
> пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev <zaleslaw.sin@gmail.com>:
> >
> > Hi, Igniters
> > I've started the work on the Spark 2.4 support
> >
> > We started the discussion here, in
> > https://issues.apache.org/jira/browse/IGNITE-12054
> >
> > The Spark internals were totally refactored between 2.3 and 2.4 versions,
> > main changes touches
> >
> >    - External catalog and listeners refactoring
> >    - Changes of HAVING operator semantic support
> >    - Push-down NULL filters generation in JOIN plans
> >    - minor changes in Plan Generation that should be adopted in our
> >    integration module
> >
> > I propose the initial solution here via creation of new module spark-2.4
> > here https://issues.apache.org/jira/browse/IGNITE-12247 and addition of
> new
> > profile spark-2.4 (to avoid possible clashes with another spark versions)
> >
> > Also I've transformed ticket to an Umbrella ticket and created a few
> > tickets for muted tests (around 7 from 211 tests are muted now)
> >
> > Please, if somebody interested in it, make an initial review of modular
> > ignite structure and changes (without deep diving into Spark code).
> >
> > And yes, the proposed code is a copy-paste of spark-ignite module with a
> > few fixes
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message