spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1911) Warn users if their assembly jars are not built with Java 6
Date Mon, 20 Apr 2015 09:51:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502589#comment-14502589
] 

Steve Loughran commented on SPARK-1911:
---------------------------------------

This doesn't fix the problem, merely documents it.

It should be doable by using Ant's <zip> task, which doesn't use the JDK zip routines.
The assembly would be unzipped first, then zipped with zip63 option set to never

see [https://ant.apache.org/manual/Tasks/zip.html]



> Warn users if their assembly jars are not built with Java 6
> -----------------------------------------------------------
>
>                 Key: SPARK-1911
>                 URL: https://issues.apache.org/jira/browse/SPARK-1911
>             Project: Spark
>          Issue Type: Bug
>          Components: Documentation
>    Affects Versions: 1.1.0
>            Reporter: Andrew Or
>            Assignee: Sean Owen
>             Fix For: 1.2.2, 1.3.0
>
>
> The root cause of the problem is detailed in: https://issues.apache.org/jira/browse/SPARK-1520.
> In short, an assembly jar built with Java 7+ is not always accessible by Python or other
versions of Java (especially Java 6). If the assembly jar is not built on the cluster itself,
this problem may manifest itself in strange exceptions that are not trivial to debug. This
is an issue especially for PySpark on YARN, which relies on the python files included within
the assembly jar.
> Currently we warn users only in make-distribution.sh, but most users build the jars directly.
At the very least we need to emphasize this in the docs (currently missing entirely). The
next step is to add a warning prompt in the mvn scripts whenever Java 7+ is detected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message