spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Russell Jurney (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1138) Spark 0.9.0 does not work with Hadoop / HDFS
Date Tue, 29 Jul 2014 01:53:40 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077270#comment-14077270
] 

Russell Jurney commented on SPARK-1138:
---------------------------------------

I built spark master with 'sbt/sbt assembly publish-local' and had issues with my hadoop version,
which is CDH 4.4. 

Then I built with CDH 4.4, via: 'SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.4.0 sbt/sbt assembly
publish-local'. Note: I did not clean. I saw this issue. This is with Spark trunk when the
released version is 1.0.1. 

Then I cleaned and rebuilt. The issue persists. 

What should I do?


> Spark 0.9.0 does not work with Hadoop / HDFS
> --------------------------------------------
>
>                 Key: SPARK-1138
>                 URL: https://issues.apache.org/jira/browse/SPARK-1138
>             Project: Spark
>          Issue Type: Bug
>            Reporter: Sam Abeyratne
>
> UPDATE: This problem is certainly related to trying to use Spark 0.9.0 and the latest
cloudera Hadoop / HDFS in the same jar.  It seems no matter how I fiddle with the deps, the
do not play nice together.
> I'm getting a java.util.concurrent.TimeoutException when trying to create a spark context
with 0.9.  I cannot, whatever I do, change the timeout.  I've tried using System.setProperty,
the SparkConf mechanism of creating a SparkContext and the -D flags when executing my jar.
 I seem to be able to run simple jobs from the spark-shell OK, but my more complicated jobs
require external libraries so I need to build jars and execute them.
> Some code that causes this:
> println("Creating config")
>     val conf = new SparkConf()
>       .setMaster(clusterMaster)
>       .setAppName("MyApp")
>       .setSparkHome(sparkHome)
>       .set("spark.akka.askTimeout", parsed.getOrElse(timeouts, "100"))
>       .set("spark.akka.timeout", parsed.getOrElse(timeouts, "100"))
>     println("Creating sc")
>     implicit val sc = new SparkContext(conf)
> The output:
> Creating config
> Creating sc
> log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> [ERROR] [02/26/2014 11:05:25.491] [main] [Remoting] Remoting error: [Startup timed out]
[
> akka.remote.RemoteTransportException: Startup timed out
> 	at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
> 	at akka.remote.Remoting.start(Remoting.scala:191)
> 	at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> 	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> 	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> 	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> 	at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> 	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> 	at com.adbrain.accuracy.EvaluateAdtruthIDs$.main(EvaluateAdtruthIDs.scala:40)
> 	at com.adbrain.accuracy.EvaluateAdtruthIDs.main(EvaluateAdtruthIDs.scala)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
> 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> 	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.result(package.scala:107)
> 	at akka.remote.Remoting.start(Remoting.scala:173)
> 	... 11 more
> ]
> Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after
[10000 milliseconds]
> 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> 	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.result(package.scala:107)
> 	at akka.remote.Remoting.start(Remoting.scala:173)
> 	at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> 	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> 	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> 	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> 	at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> 	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> 	at com.adbrain.accuracy.EvaluateAdtruthIDs$.main(EvaluateAdtruthIDs.scala:40)
> 	at com.adbrain.accuracy.EvaluateAdtruthIDs.main(EvaluateAdtruthIDs.scala)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message