ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denis Magda (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-3710) Upgrade ignite-spark module to Spark 2.0
Date Sun, 11 Dec 2016 06:48:58 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739213#comment-15739213

Denis Magda commented on IGNITE-3710:

[~vkulichenko], please review the changes which are in branch IGNITE-3710.

[~avinogradov], I failed to check my changes on TeamCity [1] because get the following exception
all the time
[09:05:42]W:	 [Step 5/6] [ERROR] error: missing or invalid dependency detected while loading
class file 'SparkContext.class'.

The exception is obviously caused by this maven related issue

[09:05:38]W:	 [Step 5/6] [WARNING] Invalid POM for org.apache.spark:spark-sql_2.11:jar:2.0.2,
transitive dependencies (if any) will not be available, enable debug logging for more details:
Some problems were encountered while processing the POMs:
[09:05:38]W:	 [Step 5/6] [ERROR] 'dependencyManagement.dependencies.dependency.groupId' for
${jline.groupid}:jline:jar with value '${jline.groupid}' does not match a valid id pattern.
@ org.apache.spark:spark-parent_2.11:2.0.2, /home/teamcity/.m2/repository/org/apache/spark/spark-parent_2.11/2.0.2/spark-parent_2.11-2.0.2.pom,
line 1838, column 18

I didn't manage to fix this maven related issue on TeamCity. However, everything works perfectly
fine and the test suite passes on my local machine (even if I clean .m2 completely).

Could you take a look and suggest how to get through this? I'm attaching the build logs in
the debug mode.

[1] http://ci.ignite.apache.org/viewLog.html?buildId=387310&tab=buildResultsDiv&buildTypeId=IgniteTests_IgniteRdd

> Upgrade ignite-spark module to Spark 2.0
> ----------------------------------------
>                 Key: IGNITE-3710
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3710
>             Project: Ignite
>          Issue Type: Improvement
>          Components: Ignite RDD
>    Affects Versions: 1.7
>            Reporter: Valentin Kulichenko
>            Assignee: Denis Magda
> Currently {{ignite-spark}} depends on Spark 1.5.2 and fails with 2.0 with this exception:
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging 
>         at java.lang.ClassLoader.defineClass1(Native Method) 
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:763) 
>         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) 
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73) 
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368) 
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362) 
>         at java.security.AccessController.doPrivileged(Native Method) 
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361) 
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
>         at org.apache.ignite.spark.JavaIgniteContext.<init>(JavaIgniteContext.scala:42)

>         at client.SparkIgniteClient.main(SparkIgniteClient.java:75) 
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

>         at java.lang.reflect.Method.invoke(Method.java:498) 
>         at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) 
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging 
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
> {noformat}
> Need to investigate if we can upgrade without breaking compatibility with old versions.

This message was sent by Atlassian JIRA

View raw message