Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
Date: Sun, 11 Dec 2016 06:48:58 +0000 (UTC)
From: "Denis Magda (JIRA)" <jira@apache.org>
To: issues@ignite.apache.org
Message-ID: <JIRA.12998306.1471553032000.492749.1481438938980@Atlassian.JIRA>
In-Reply-To: <JIRA.12998306.1471553032000@Atlassian.JIRA>
References: <JIRA.12998306.1471553032000@Atlassian.JIRA> <JIRA.12998306.1471553032393@arcas>
Subject: [jira] [Commented] (IGNITE-3710) Upgrade ignite-spark module to
 Spark 2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 11 Dec 2016 06:49:01 -0000


    [ https://issues.apache.org/jira/browse/IGNITE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739213#comment-15739213 ] 

Denis Magda commented on IGNITE-3710:
-------------------------------------

[~vkulichenko], please review the changes which are in branch IGNITE-3710.

[~avinogradov], I failed to check my changes on TeamCity [1] because get the following exception all the time
{code}
[09:05:42]W:	 [Step 5/6] [ERROR] error: missing or invalid dependency detected while loading class file 'SparkContext.class'.
{code}

The exception is obviously caused by this maven related issue

{code}
[09:05:38]W:	 [Step 5/6] [WARNING] Invalid POM for org.apache.spark:spark-sql_2.11:jar:2.0.2, transitive dependencies (if any) will not be available, enable debug logging for more details: Some problems were encountered while processing the POMs:
[09:05:38]W:	 [Step 5/6] [ERROR] 'dependencyManagement.dependencies.dependency.groupId' for ${jline.groupid}:jline:jar with value '${jline.groupid}' does not match a valid id pattern. @ org.apache.spark:spark-parent_2.11:2.0.2, /home/teamcity/.m2/repository/org/apache/spark/spark-parent_2.11/2.0.2/spark-parent_2.11-2.0.2.pom, line 1838, column 18
{code}

I didn't manage to fix this maven related issue on TeamCity. However, everything works perfectly fine and the test suite passes on my local machine (even if I clean .m2 completely).

Could you take a look and suggest how to get through this? I'm attaching the build logs in the debug mode.


[1] http://ci.ignite.apache.org/viewLog.html?buildId=387310&tab=buildResultsDiv&buildTypeId=IgniteTests_IgniteRdd


> Upgrade ignite-spark module to Spark 2.0
> ----------------------------------------
>
>                 Key: IGNITE-3710
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3710
>             Project: Ignite
>          Issue Type: Improvement
>          Components: Ignite RDD
>    Affects Versions: 1.7
>            Reporter: Valentin Kulichenko
>            Assignee: Denis Magda
>
> Currently {{ignite-spark}} depends on Spark 1.5.2 and fails with 2.0 with this exception:
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging 
>         at java.lang.ClassLoader.defineClass1(Native Method) 
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:763) 
>         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) 
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73) 
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368) 
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362) 
>         at java.security.AccessController.doPrivileged(Native Method) 
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361) 
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
>         at org.apache.ignite.spark.JavaIgniteContext.<init>(JavaIgniteContext.scala:42) 
>         at client.SparkIgniteClient.main(SparkIgniteClient.java:75) 
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
>         at java.lang.reflect.Method.invoke(Method.java:498) 
>         at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) 
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging 
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
> {noformat}
> Need to investigate if we can upgrade without breaking compatibility with old versions.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)