spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peng Cheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6473) Launcher lib shouldn't try to figure out Scala version when not in dev mode
Date Tue, 05 Dec 2017 23:16:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279355#comment-16279355
] 

Peng Cheng commented on SPARK-6473:
-----------------------------------

Looks like this issue reappear at some point:

getScalaVersion() will be called anyway even if not in dev/test mode:

https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L196

It was suppressed simply because SPARK_SCALA_VERSION is always set by shell script.

Should we add a test to ensure that it never happens?

> Launcher lib shouldn't try to figure out Scala version when not in dev mode
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-6473
>                 URL: https://issues.apache.org/jira/browse/SPARK-6473
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 1.4.0
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>             Fix For: 1.4.0
>
>
> Thanks to [~nravi] for pointing this out.
> The launcher library currently always tries to figure out what's the build's scala version,
even when it's not needed. That code is only used when setting some dev options, and relies
on the layout of the build directories, so it doesn't work with the directory layout created
by make-distribution.sh.
> Right now this works on Linux because bin/load-spark-env.sh sets the Scala version explicitly,
but it would break the distribution on Windows, for example.
> Fix is pretty straight-forward.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message