Return-Path: <issues-return-113818-archive-asf-public=cust-asf.ponee.io@spark.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id C5080200B13
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Wed,  1 Jun 2016 02:23:14 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id C0A44160A44; Wed,  1 Jun 2016 00:23:14 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 0117E160A48
	for <archive-asf-public@cust-asf.ponee.io>; Wed,  1 Jun 2016 02:23:13 +0200 (CEST)
Received: (qmail 9355 invoked by uid 500); 1 Jun 2016 00:23:13 -0000
Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@spark.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@spark.apache.org>
List-Post: <mailto:issues@spark.apache.org>
List-Id: <issues.spark.apache.org>
Delivered-To: mailing list issues@spark.apache.org
Received: (qmail 9344 invoked by uid 99); 1 Jun 2016 00:23:13 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2016 00:23:13 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 03C4B2C1F5A
	for <issues@spark.apache.org>; Wed,  1 Jun 2016 00:23:13 +0000 (UTC)
Date: Wed, 1 Jun 2016 00:23:13 +0000 (UTC)
From: "Bryan Cutler (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12927917.1452059126000.344301.1464740593012@Atlassian.JIRA>
In-Reply-To: <JIRA.12927917.1452059126000@Atlassian.JIRA>
References: <JIRA.12927917.1452059126000@Atlassian.JIRA> <JIRA.12927917.1452059126932@arcas>
Subject: [jira] [Commented] (SPARK-12666) spark-shell --packages cannot load
 artifacts which are publishLocal'd by SBT
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Wed, 01 Jun 2016 00:23:14 -0000


    [ https://issues.apache.org/jira/browse/SPARK-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308950#comment-15308950 ] 

Bryan Cutler commented on SPARK-12666:
--------------------------------------

This seems like it is more of an issue with SBT publishLocal, as it publishes an ivy.xml file without a "default" conf.  Like this, for example

{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
        ...
        <configurations>
                <conf name="compile" visibility="public" description=""/>
                <conf name="runtime" visibility="public" description="" extends="compile"/>
                <conf name="test" visibility="public" description="" extends="runtime"/>
                <conf name="provided" visibility="public" description=""/>
                <conf name="optional" visibility="public" description=""/>
                <conf name="it" visibility="public" description="" extends="test"/>
                <conf name="sources" visibility="public" description=""/>
                <conf name="docs" visibility="public" description=""/>
                <conf name="pom" visibility="public" description=""/>
        </configurations>
...
{noformat}

If this file had a conf name="default", or if there were no confs defined an implicit "default" would be used.  But since it is missing, there is an error when resolving.  I think it's reasonable for Spark to expect a "default" conf to exist, but we can make it a bit more robust to handle cases like this where it doesn't exist.  I'll post a PR for this.

> spark-shell --packages cannot load artifacts which are publishLocal'd by SBT
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-12666
>                 URL: https://issues.apache.org/jira/browse/SPARK-12666
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Josh Rosen
>
> Symptom:
> I cloned the latest master of {{spark-redshift}}, then used {{sbt publishLocal}} to publish it to my Ivy cache. When I tried running {{./bin/spark-shell --packages com.databricks:spark-redshift_2.10:0.5.3-SNAPSHOT}} to load this dependency into {{spark-shell}}, I received the following cryptic error:
> {code}
> Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.databricks#spark-redshift_2.10;0.5.3-SNAPSHOT: configuration not found in com.databricks#spark-redshift_2.10;0.5.3-SNAPSHOT: 'default'. It was required from org.apache.spark#spark-submit-parent;1.0 default]
> 	at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
> 	at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> I think the problem here is that Spark is declaring a dependency on the spark-redshift artifact using the {{default}} Ivy configuration. Based on my admittedly limited understanding of Ivy, the default configuration will be the only configuration defined in an Ivy artifact if that artifact defines no other configurations. Thus, for Maven artifacts I think the default configuration will end up mapping to Maven's regular JAR dependency (i.e. Maven artifacts don't declare Ivy configurations so they implicitly have the {{default}} configuration) but for Ivy artifacts I think we can run into trouble when loading artifacts which explicitly define their own configurations, since those artifacts might not have a configuration named {{default}}.
> I spent a bit of time playing around with the SparkSubmit code to see if I could fix this but wasn't able to completely resolve the issue.
> /cc [~brkyvz] (ping me offline and I can walk you through the repo in person, if you'd like)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org