spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21403) Cluster mode doesn't work with --packages [Mesos]
Date Thu, 13 Jul 2017 14:25:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stavros Kontopoulos updated SPARK-21403:
----------------------------------------
    Summary: Cluster mode doesn't work with --packages [Mesos]  (was: Cluster mode doesn't
work with --packages)

> Cluster mode doesn't work with --packages [Mesos]
> -------------------------------------------------
>
>                 Key: SPARK-21403
>                 URL: https://issues.apache.org/jira/browse/SPARK-21403
>             Project: Spark
>          Issue Type: Bug
>          Components: Mesos
>    Affects Versions: 1.3.0
>            Reporter: Stavros Kontopoulos
>
> From the mailing list:
> {quote}
> Another problem I ran into that you also might is that --packages doesn't
> work with --deploy-mode cluster.  It downloads the packages to a temporary
> location on the node running spark-submit, then passes those paths to the
> node that is running the Driver, but since that isn't the same machine, it
> can't find anything and fails.  The driver process *should* be the one
> doing the downloading, but it isn't. I ended up having to create a fat JAR
> with all of the dependencies to get around that one.
> {quote}
> The problem is that we currently don't upload jars to the cluster. It seems to fix this
we either (1) do upload jars, or (2) just run the packages code on the driver side. I slightly
prefer (2) because it's simpler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message