ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Zinoviev <zaleslaw....@gmail.com>
Subject Re: The Spark 2.4 support
Date Mon, 18 Nov 2019 18:05:47 GMT
Right, a few tests from 200 are failed due to known issue and couldnt be
fixed immediately, related to rare cases. These tests are copies of 2.3
tests and part of them could have no meaning for 2.4 due to Spark changed
behaviour.

пн, 18 нояб. 2019 г., 19:42 Denis Magda <dmagda@apache.org>:

> Alexey,
>
> Please help to understand what it means that 2.4 integration supports "95%
> of tests of 2.3". Does it mean that 5% of existing tests are failing and,
> basically, need to be fixed?
>
> -
> Denis
>
>
> On Mon, Nov 18, 2019 at 6:52 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
> wrote:
>
> > Dear Nikolay Izhikov, I've recreated the PR for 2.4 initial support
> >
> > The last commit
> >
> >
> https://github.com/apache/ignite/pull/7058/commits/60386802299deedc6ed60bf4736e922201a67fb8
> > contains
> > real changes from Spark 2.3
> >
> > I suggest to merge to master this initial solution with 95% support of
> > Spark 2.4 and continue work on known issues listed in JIRA
> >
> > This solution supports the new Spark version for all examples and 95% of
> > tests of 2.3.
> >
> > вт, 1 окт. 2019 г. в 08:48, Ivan Pavlukhin <vololo100@gmail.com>:
> >
> > > Alexey, Nikolay,
> > >
> > > Thank you for sharing details!
> > >
> > > вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev <zaleslaw.sin@gmail.com>:
> > > >
> > > > Great talk and paper, I've learnt it last year
> > > >
> > > > пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov <nizhikov@apache.org>:
> > > >
> > > > > Yes, I can :)
> > > > >
> > > > > В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет:
> > > > > > Nikolay,
> > > > > >
> > > > > > Would you be able to review the changes? I'm not sure there
is a
> > > better
> > > > > candidate for now.
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov <
> > > nizhikov@apache.org>
> > > > > wrote:
> > > > > > > Hello, Ivan.
> > > > > > >
> > > > > > > I had a talk about internals of Spark integration in Ignite.
> > > > > > > It answers on question why we should use Spark internals.
> > > > > > >
> > > > > > > You can take a look at my meetup talk(in Russian) [1] or
read
> an
> > > > > article if you prefer text [2].
> > > > > > >
> > > > > > > [1] https://www.youtube.com/watch?v=CzbAweNKEVY
> > > > > > > [2] https://habr.com/ru/company/sberbank/blog/427297/
> > > > > > >
> > > > > > > В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev пишет:
> > > > > > > > Yes, as I understand it uses Spark internals from
the first
> > > commit)))
> > > > > > > > The reason - we take Spark SQL query execution plan
and try
> to
> > > > > execute it
> > > > > > > > on Ignite cluster
> > > > > > > > Also we inherit a lot of Developer API related classes
that
> > > could be
> > > > > > > > unstable. Spark has no good point for extension and
this is a
> > > reason
> > > > > why we
> > > > > > > > should go deeper
> > > > > > > >
> > > > > > > > пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin
<
> > > vololo100@gmail.com>:
> > > > > > > >
> > > > > > > > > Hi Alexey,
> > > > > > > > >
> > > > > > > > > As an external watcher very far from Ignite Spark
> > integration I
> > > > > would
> > > > > > > > > like to ask a humble question for my understanding.
Why
> this
> > > > > > > > > integration uses Spark internals? Is it a common
approach
> for
> > > > > > > > > integrating with Spark?
> > > > > > > > >
> > > > > > > > > пн, 30 сент. 2019 г. в 16:17, Alexey
Zinoviev <
> > > > > zaleslaw.sin@gmail.com>:
> > > > > > > > > >
> > > > > > > > > > Hi, Igniters
> > > > > > > > > > I've started the work on the Spark 2.4 support
> > > > > > > > > >
> > > > > > > > > > We started the discussion here, in
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054
> > > > > > > > > >
> > > > > > > > > > The Spark internals were totally refactored
between 2.3
> and
> > > 2.4
> > > > > versions,
> > > > > > > > > > main changes touches
> > > > > > > > > >
> > > > > > > > > >    - External catalog and listeners refactoring
> > > > > > > > > >    - Changes of HAVING operator semantic
support
> > > > > > > > > >    - Push-down NULL filters generation in
JOIN plans
> > > > > > > > > >    - minor changes in Plan Generation that
should be
> > adopted
> > > in
> > > > > our
> > > > > > > > > >    integration module
> > > > > > > > > >
> > > > > > > > > > I propose the initial solution here via
creation of new
> > > module
> > > > > spark-2.4
> > > > > > > > > > here https://issues.apache.org/jira/browse/IGNITE-12247
> > and
> > > > > addition of
> > > > > > > > >
> > > > > > > > > new
> > > > > > > > > > profile spark-2.4 (to avoid possible clashes
with another
> > > spark
> > > > > versions)
> > > > > > > > > >
> > > > > > > > > > Also I've transformed ticket to an Umbrella
ticket and
> > > created a
> > > > > few
> > > > > > > > > > tickets for muted tests (around 7 from 211
tests are
> muted
> > > now)
> > > > > > > > > >
> > > > > > > > > > Please, if somebody interested in it, make
an initial
> > review
> > > of
> > > > > modular
> > > > > > > > > > ignite structure and changes (without deep
diving into
> > Spark
> > > > > code).
> > > > > > > > > >
> > > > > > > > > > And yes, the proposed code is a copy-paste
of
> spark-ignite
> > > > > module with a
> > > > > > > > > > few fixes
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best regards,
> > > > > > > > > Ivan Pavlukhin
> > > > > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message