spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pedrorodriguez <>
Subject sbt publish-local fails, missing spark-network-common
Date Fri, 21 Nov 2014 03:53:47 GMT
I am developing an application which calls into Spark MLlib I am working on (LDA). To do so,
I am linking Spark locally in the application and using sbt assembly/publish-local in the
spark directory. 

When I run sbt assembly in my application I get the following error: 
$ sbt assembly 
[info] Loading global plugins from /Users/pedro/.sbt/0.13/plugins 
[info] Loading project definition from /Users/pedro/Documents/Code/nips-lda/project 
[info] Set current project to nips-lda (in build file:/Users/pedro/Documents/Code/nips-lda/) 
[info] Updating {file:/Users/pedro/Documents/Code/nips-lda/}nips-lda... 
[info] Resolving org.apache.spark#spark-network-common_2.10;1.3.0-SNAPSHOT ... 
[warn] module not found: org.apache.spark#spark-network-common_2.10;1.3.0-SNAPSHOT 
[warn] ==== local: tried 
[warn]   /Users/pedro/.ivy2/local/org.apache.spark/spark-network-common_2.10/1.3.0-SNAPSHOT/ivys/ivy.xml 
[warn] ==== public: tried 
[warn] ==== Typesafe: tried 
[warn] ==== Spray: tried 
[info] Resolving org.fusesource.jansi#jansi;1.4 ... 
[warn] :::::::::::::::::::::::::::::::::::::::::::::: 
[warn] ::          UNRESOLVED DEPENDENCIES         :: 
[warn] :::::::::::::::::::::::::::::::::::::::::::::: 
[warn] :: org.apache.spark#spark-network-common_2.10;1.3.0-SNAPSHOT: not found 
[warn] :::::::::::::::::::::::::::::::::::::::::::::: 
[warn] Note: Unresolved dependencies path: 
[warn] org.apache.spark:spark-network-common_2.10:1.3.0-SNAPSHOT 
[warn]  +- org.apache.spark:spark-network-shuffle_2.10:1.3.0-SNAPSHOT 
[warn]  +- org.apache.spark:spark-core_2.10:1.3.0-SNAPSHOT (/Users/pedro/Documents/Code/nips-lda/build.sbt#L15-26) 
[warn]  +- org.apache.spark:spark-catalyst_2.10:1.3.0-SNAPSHOT 
[warn]  +- org.apache.spark:spark-sql_2.10:1.3.0-SNAPSHOT 
[warn]  +- org.apache.spark:spark-mllib_2.10:1.3.0-SNAPSHOT (/Users/pedro/Documents/Code/nips-lda/build.sbt#L15-26) 
[warn]  +- edu.berkeley.cs.amplab:nips-lda_2.10:0.1 
sbt.ResolveException: unresolved dependency: org.apache.spark#spark-network-common_2.10;1.3.0-SNAPSHOT:
not found 
        at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243) 
        at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158) 

Inspecting the sbt publish-local logs and ~/.ivy2/local/org.apache.spark it looks like spark-network-common
is not getting published, which causes this error in build time. I am rebased on upstream/master
recently and it looks like this was recently added. Inspecting the pom.xml in spark, it looks
like submodules look in order, same with project/SparkBuild.scala. 

Any suggestions on how to fix this? 

P.S. is javadoc compilation broken atm?


View this message in context:
Sent from the Apache Spark Developers List mailing list archive at
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message