spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastien Rainville (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2022) Spark 1.0.0 is failing if mesos.coarse set to true
Date Tue, 17 Jun 2014 20:07:16 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034294#comment-14034294
] 

Sebastien Rainville commented on SPARK-2022:
--------------------------------------------

I'm seeing the same behavior when trying to set spark.executor.extraLibraryPath:

in conf/spark-defaults.conf:

spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native

the error message in stderr:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0617 16:00:55.592289 27091 fetcher.cpp:73] Fetching URI 'hdfs://ca1-dcc1-0071:9200/user/sebastien/spark-1.0.0-bin-cdh4-sebr.tgz'
I0617 16:00:55.592428 27091 fetcher.cpp:99] Downloading resource from 'hdfs://ca1-dcc1-0071:9200/user/sebastien/spark-1.0.0-bin-cdh4-sebr.tgz'
to '/u05/app/mesos/work/slaves/201311011608-1369465866-5050-9189-86/frameworks/20140416-011500-1369465866-5050-26096-0449/executors/9/runs/ba87d7b6-56c1-4892-9ed8-18fa8f8364d2/spark-1.0.0-bin-cdh4-sebr.tgz'
I0617 16:01:05.170714 27091 fetcher.cpp:61] Extracted resource '/u05/app/mesos/work/slaves/201311011608-1369465866-5050-9189-86/frameworks/20140416-011500-1369465866-5050-26096-0449/executors/9/runs/ba87d7b6-56c1-4892-9ed8-18fa8f8364d2/spark-1.0.0-bin-cdh4-sebr.tgz'
into '/u05/app/mesos/work/slaves/201311011608-1369465866-5050-9189-86/frameworks/20140416-011500-1369465866-5050-26096-0449/executors/9/runs/ba87d7b6-56c1-4892-9ed8-18fa8f8364d2'
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0617 16:01:06.105363 27166 exec.cpp:131] Version: 0.18.0
I0617 16:01:06.112191 27175 exec.cpp:205] Executor registered on slave 201311011608-1369465866-5050-9189-86
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Exception in thread "main" java.lang.NumberFormatException: For input string: "ca1-dcc1-0106.lab.mtl"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:492)
	at java.lang.Integer.parseInt(Integer.java:527)
	at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
	at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:135)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

here is the command in stdout:


Registered executor on ca1-dcc1-0106.lab.mtl
Starting task 9
Forked command at 27178
sh -c 'cd spark-1*; ./bin/spark-class org.apache.spark.executor.CoarseGrainedExecutorBackend
-Djava.library.path=/usr/lib/hadoop/lib/native akka.tcp://spark@ca1-dcc1-0071.lab.mtl:32789/user/CoarseGrainedScheduler
201311011608-1369465866-5050-9189-86 ca1-dcc1-0106.lab.mtl 1'
Command exited with status 1 (pid: 27178)


> Spark 1.0.0 is failing if mesos.coarse set to true
> --------------------------------------------------
>
>                 Key: SPARK-2022
>                 URL: https://issues.apache.org/jira/browse/SPARK-2022
>             Project: Spark
>          Issue Type: Bug
>          Components: Mesos
>    Affects Versions: 1.0.0
>            Reporter: Marek Wiewiorka
>            Priority: Critical
>
> more stderr
> -------------------
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0603 16:07:53.721132 61192 exec.cpp:131] Version: 0.18.2
> I0603 16:07:53.725230 61200 exec.cpp:205] Executor registered on slave 201405220917-134217738-5050-27119-0
> Exception in thread "main" java.lang.NumberFormatException: For input string: "sparkseq003.cloudapp.net"
>         at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Integer.parseInt(Integer.java:492)
>         at java.lang.Integer.parseInt(Integer.java:527)
>         at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
>         at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
>         at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:135)
>         at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> more stdout
> -----------------------
> Registered executor on sparkseq003.cloudapp.net
> Starting task 5
> Forked command at 61202
> sh -c '"/home/mesos/spark-1.0.0/bin/spark-class" org.apache.spark.executor.CoarseGrainedExecutorBackend
-Dspark.mesos.coarse=true akka.tcp://spark@sparkseq001.cloudapp.net:40312/user/CoarseG
> rainedScheduler 201405220917-134217738-5050-27119-0 sparkseq003.cloudapp.net 4'
> Command exited with status 1 (pid: 61202)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message