ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Izhikov <nizhi...@apache.org>
Subject Re: Ignite-Spark integration meeting
Date Wed, 23 Oct 2019 15:57:47 GMT
The only issue with the Spark integration tests - they should be runned under JDK8.
That's all.

I wrote about it recently [1]

[1] http://apache-ignite-developers.2346864.n4.nabble.com/Spark-examples-on-TC-td43591.html

В Ср, 23/10/2019 в 17:46 +0200, Alexey Zinoviev пишет:
> Good point, now they are not in the Best state, we have a few ideas how to
> fix, will discuss it later here in this thread.
> 
> If you have any objection write it, but my position - creation special
> Spark Visa on TC, like for ML, maybe two for different Spark versions, and
> all commits to Spark will require this Visa.
> 
> 
> ср, 23 окт. 2019 г., 16:25 Ivan Pavlukhin <vololo100@gmail.com>:
> 
> > Actually, I meant running tests on TeamCity.
> > 
> > ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <zaleslaw.sin@gmail.com>:
> > > 
> > > What kind of testing do you mean?  JUnit, integration via example or
> > > something else?
> > > 
> > > ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <vololo100@gmail.com>:
> > > 
> > > > Correction.
> > > > 
> > > > How are you going to TEST integrations with different Spark versions?
> > > > 
> > > > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vololo100@gmail.com>:
> > > > > 
> > > > > Alexey,
> > > > > 
> > > > > Thank you for sharing the summary!
> > > > > 
> > > > > Small question. How are you going to integrations with different
> > 
> > Spark
> > > > versions?
> > > > > 
> > > > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <dmagda@apache.org>:
> > > > > > 
> > > > > > > 
> > > > > > >  1. We are going to release Apache Ignite 2.8 with limited
> > 
> > support of
> > > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > > 
> > > > > > 
> > > > > > What does limited support mean? Is it about regressions (something
> > 
> > that
> > > > > > worked before but will fail) or capabilities unsupported even
for
> > 
> > the
> > > > > > presently released version of the integration?
> > > > > > 
> > > > > > -
> > > > > > Denis
> > > > > > 
> > > > > > 
> > > > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > > > 
> > > > zaleslaw.sin@gmail.com>
> > > > > > wrote:
> > > > > > 
> > > > > > > The meeting notes and results of discussion are next:
> > > > > > > 
> > > > > > >    1. We are going to release Apache Ignite 2.8 with limited
> > 
> > support
> > > > of
> > > > > > >    Spark 2.4.4 (known issues are listed here
> > > > > > >    https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > > >    2. The AI 2.8 will support both, Spark 2.3 and Spark
2.4.4
> > 
> > with
> > > > known
> > > > > > >    fixed bugs
> > > > > > >    3. We will create the developer-profiles spark_2_3 and
> > 
> > spark_2_4
> > > > > > >    separately from current scala profile
> > > > > > >    4. We are ready to move the Ignite-Spark integration
to the
> > > > 
> > > > separate
> > > > > > >    module after release 2.8 with support of most actual
Spark
> > > > 
> > > > versions and
> > > > > > >    with independent release cycle
> > > > > > >    5. We will work together on the next release roadmap
with
> > 
> > Nikolay
> > > > and
> > > > > > >    all interested igniters to improve the current solution
(add
> > 
> > new
> > > > > > > features
> > > > > > >    as thin client support, working with caches via binary
object
> > 
> > and
> > > > so on)
> > > > > > >    6. Nikolay is ready to help with the review of tickets
as an
> > > > 
> > > > author of
> > > > > > >    this component
> > > > > > > 
> > > > > > > Thanks
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <
> > 
> > zaleslaw.sin@gmail.com
> > > > > :
> > > > > > > 
> > > > > > > > Yes, let's use the Russian for short discussion, the
meeting
> > 
> > notes
> > > > will
> > > > > > > be
> > > > > > > > presented in English
> > > > > > > > 
> > > > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov
<
> > 
> > nizhikov@apache.org
> > > > > :
> > > > > > > > 
> > > > > > > > > Hello, Alexey.
> > > > > > > > > 
> > > > > > > > > I'm ready to take part in the meeting.
> > > > > > > > > It will be in Russian, isn't it?
> > > > > > > > > 
> > > > > > > > > В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev
пишет:
> > > > > > > > > > Hi, Igniters, and especially Nikolay Izhikov
> > > > > > > > > > 
> > > > > > > > > > I suggest to arrange the meeting via ASF-slack
in
> > > > > > > > > 
> > > > > > > > > #ignite-spark-integration
> > > > > > > > > > channel at 17 October, in 16 MSK Time.
> > > > > > > > > > The language of the meeting: Russian
> > > > > > > > > > 
> > > > > > > > > > *The topics to discuss:*
> > > > > > > > > > 
> > > > > > > > > >    - Review of next tickets
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > >    1. [Spark][Bug] IgniteSpark integration
forget to close
> > 
> > the
> > > > > > > > > >       IgniteContext and stops the client
node in case if
> > 
> > error
> > > > during
> > > > > > > > > >       PairFunction logic
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > > > > > >       2. [Spark] Add initial support of
Spark 2.4
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > > > > > >       3. [Spark] Add joins to Spark Dataframe
examples
> > > > > > > > > >       <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > >    - Known issues of migration to 2.4 version
> > > > > > > > > >    - Organization of multiple version support
for Apache
> > 
> > Spark
> > > > > > > > > > 
> > > > > > > > > > P.S. meeting notes will be shared with the
community
> > > > > > > > > > P.P.S Nikolay, if you have any objections
about the time or
> > > > 
> > > > topics,
> > > > > > > > > please
> > > > > > > > > > let me know
> > > > > 
> > > > > 
> > > > > 
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > 
> > > > 
> > > > 
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > > 
> > 
> > 
> > 
> > --
> > Best regards,
> > Ivan Pavlukhin
> > 

Mime
View raw message