spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-24825) [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure
Date Tue, 17 Jul 2018 21:32:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547128#comment-16547128
] 

Stavros Kontopoulos edited comment on SPARK-24825 at 7/17/18 9:31 PM:
----------------------------------------------------------------------

As I mentioned elsewhere one thing that worked for me is modify ./dev/make-distribution.sh
to do a clean install instead of clean package. But not sure for the long term for this, although
you could purge deps.

Test suite has the following spark deps:

mvn -f pom.xml dependency:resolve | grep spark

[INFO] org.apache.spark:spark-kvstore_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-unsafe_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-launcher_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-tags_2.11:jar:2.4.0-SNAPSHOT:compile
 [INFO] org.apache.spark:spark-tags_2.11:test-jar:tests:2.4.0-SNAPSHOT:compile
 [INFO] org.spark-project.spark:unused:jar:1.0.0:compile
 [INFO] org.apache.spark:spark-network-common_2.11:jar:2.4.0-SNAPSHOT:compile


was (Author: skonto):
As I mentioned elsewhere one thing that worked for me is modify ./dev/make-distribution.sh
to do an clean install instead of clean package. But not sure for the long term.

Test suite has the following spark deps:

mvn -f pom.xml dependency:resolve | grep spark

[INFO] org.apache.spark:spark-kvstore_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-unsafe_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-launcher_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:test-jar:tests:2.4.0-SNAPSHOT:compile
[INFO] org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] org.apache.spark:spark-network-common_2.11:jar:2.4.0-SNAPSHOT:compile

> [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24825
>                 URL: https://issues.apache.org/jira/browse/SPARK-24825
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes, Tests
>    Affects Versions: 2.4.0
>            Reporter: Matt Cheah
>            Priority: Critical
>
> The Kubernetes integration tests will currently fail if maven installation is not performed
first, because the integration test build believes it should be pulling the Spark parent artifact
from maven central. However, this is incorrect because the integration test should be building
the Spark parent pom as a dependency in the multi-module build, and the integration test should
just use the dynamically built artifact. Or to put it another way, the integration test builds
should never be pulling Spark dependencies from maven central.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message