ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: The Spark 2.4 support
Date Mon, 18 Nov 2019 16:41:59 GMT
Alexey,

Please help to understand what it means that 2.4 integration supports "95%
of tests of 2.3". Does it mean that 5% of existing tests are failing and,
basically, need to be fixed?

-
Denis


On Mon, Nov 18, 2019 at 6:52 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
wrote:

> Dear Nikolay Izhikov, I've recreated the PR for 2.4 initial support
>
> The last commit
>
> https://github.com/apache/ignite/pull/7058/commits/60386802299deedc6ed60bf4736e922201a67fb8
> contains
> real changes from Spark 2.3
>
> I suggest to merge to master this initial solution with 95% support of
> Spark 2.4 and continue work on known issues listed in JIRA
>
> This solution supports the new Spark version for all examples and 95% of
> tests of 2.3.
>
> вт, 1 окт. 2019 г. в 08:48, Ivan Pavlukhin <vololo100@gmail.com>:
>
> > Alexey, Nikolay,
> >
> > Thank you for sharing details!
> >
> > вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev <zaleslaw.sin@gmail.com>:
> > >
> > > Great talk and paper, I've learnt it last year
> > >
> > > пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov <nizhikov@apache.org>:
> > >
> > > > Yes, I can :)
> > > >
> > > > В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет:
> > > > > Nikolay,
> > > > >
> > > > > Would you be able to review the changes? I'm not sure there is a
> > better
> > > > candidate for now.
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov <
> > nizhikov@apache.org>
> > > > wrote:
> > > > > > Hello, Ivan.
> > > > > >
> > > > > > I had a talk about internals of Spark integration in Ignite.
> > > > > > It answers on question why we should use Spark internals.
> > > > > >
> > > > > > You can take a look at my meetup talk(in Russian) [1] or read
an
> > > > article if you prefer text [2].
> > > > > >
> > > > > > [1] https://www.youtube.com/watch?v=CzbAweNKEVY
> > > > > > [2] https://habr.com/ru/company/sberbank/blog/427297/
> > > > > >
> > > > > > В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev пишет:
> > > > > > > Yes, as I understand it uses Spark internals from the first
> > commit)))
> > > > > > > The reason - we take Spark SQL query execution plan and
try to
> > > > execute it
> > > > > > > on Ignite cluster
> > > > > > > Also we inherit a lot of Developer API related classes
that
> > could be
> > > > > > > unstable. Spark has no good point for extension and this
is a
> > reason
> > > > why we
> > > > > > > should go deeper
> > > > > > >
> > > > > > > пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin <
> > vololo100@gmail.com>:
> > > > > > >
> > > > > > > > Hi Alexey,
> > > > > > > >
> > > > > > > > As an external watcher very far from Ignite Spark
> integration I
> > > > would
> > > > > > > > like to ask a humble question for my understanding.
Why this
> > > > > > > > integration uses Spark internals? Is it a common approach
for
> > > > > > > > integrating with Spark?
> > > > > > > >
> > > > > > > > пн, 30 сент. 2019 г. в 16:17, Alexey Zinoviev
<
> > > > zaleslaw.sin@gmail.com>:
> > > > > > > > >
> > > > > > > > > Hi, Igniters
> > > > > > > > > I've started the work on the Spark 2.4 support
> > > > > > > > >
> > > > > > > > > We started the discussion here, in
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054
> > > > > > > > >
> > > > > > > > > The Spark internals were totally refactored between
2.3 and
> > 2.4
> > > > versions,
> > > > > > > > > main changes touches
> > > > > > > > >
> > > > > > > > >    - External catalog and listeners refactoring
> > > > > > > > >    - Changes of HAVING operator semantic support
> > > > > > > > >    - Push-down NULL filters generation in JOIN
plans
> > > > > > > > >    - minor changes in Plan Generation that should
be
> adopted
> > in
> > > > our
> > > > > > > > >    integration module
> > > > > > > > >
> > > > > > > > > I propose the initial solution here via creation
of new
> > module
> > > > spark-2.4
> > > > > > > > > here https://issues.apache.org/jira/browse/IGNITE-12247
> and
> > > > addition of
> > > > > > > >
> > > > > > > > new
> > > > > > > > > profile spark-2.4 (to avoid possible clashes
with another
> > spark
> > > > versions)
> > > > > > > > >
> > > > > > > > > Also I've transformed ticket to an Umbrella ticket
and
> > created a
> > > > few
> > > > > > > > > tickets for muted tests (around 7 from 211 tests
are muted
> > now)
> > > > > > > > >
> > > > > > > > > Please, if somebody interested in it, make an
initial
> review
> > of
> > > > modular
> > > > > > > > > ignite structure and changes (without deep diving
into
> Spark
> > > > code).
> > > > > > > > >
> > > > > > > > > And yes, the proposed code is a copy-paste of
spark-ignite
> > > > module with a
> > > > > > > > > few fixes
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Ivan Pavlukhin
> > > > > > > >
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message