spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21159) Cluster mode, driver throws connection refused exception submitted by SparkLauncher
Date Wed, 21 Jun 2017 02:34:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056872#comment-16056872
] 

Marcelo Vanzin commented on SPARK-21159:
----------------------------------------

standalone cluster mode applications are not supported through {{startApplication}} yet. See
SPARK-11033.

I don't think they should fail, so I'm not going to make this a duplicate yet, in case there's
something to fix here.

> Cluster mode, driver throws connection refused exception submitted by SparkLauncher
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-21159
>                 URL: https://issues.apache.org/jira/browse/SPARK-21159
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, Spark Submit
>    Affects Versions: 2.1.0
>         Environment: Server A-Master
> Server B-Slave
>            Reporter: niefei
>
> When an spark application submitted by SparkLauncher#startApplication method, this will
get a SparkAppHandle. In the test environment, the launcher runs on server A, if it runs in
Client mode, everything is ok. In cluster mode, the launcher will run on Server A, and the
driver will be run on Server B, in this scenario, when initialize SparkContext, a LauncherBackend
will try to connect to the launcher application via specified port and ip address. the problem
is the implementation of LauncherBackend uses loopback ip to connect which is 127.0.0.1. this
will cause the connection refused as server B never ran the launcher. 
> The expected behavior is the LauncherBackend should use Server A's Ip address to connect
for reporting the running status.
> Below is the stacktrace:
> 17/06/20 17:24:37 ERROR SparkContext: Error initializing SparkContext.
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at java.net.Socket.connect(Socket.java:538)
> 	at java.net.Socket.<init>(Socket.java:434)
> 	at java.net.Socket.<init>(Socket.java:244)
> 	at org.apache.spark.launcher.LauncherBackend.connect(LauncherBackend.scala:43)
> 	at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.start(StandaloneSchedulerBackend.scala:60)
> 	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
> 	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
> 	at scala.Option.getOrElse(Option.scala:121)
> 	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.executeSparkJob(AbstractCommonSparkTask.scala:91)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.runSparkJob(AbstractCommonSparkTask.scala:25)
> 	at com.asura.grinder.datatask.main.TaskMain$.main(TaskMain.scala:61)
> 	at com.asura.grinder.datatask.main.TaskMain.main(TaskMain.scala)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
> 	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> 17/06/20 17:24:37 INFO SparkUI: Stopped Spark web UI at http://172.25.108.62:4040
> 17/06/20 17:24:37 INFO StandaloneSchedulerBackend: Shutting down all executors
> 17/06/20 17:24:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor
to shut down
> 17/06/20 17:24:37 ERROR Utils: Uncaught exception in thread main
> java.lang.NullPointerException
> 	at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.org$apache$spark$scheduler$cluster$StandaloneSchedulerBackend$$stop(StandaloneSchedulerBackend.scala:214)
> 	at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.stop(StandaloneSchedulerBackend.scala:116)
> 	at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:467)
> 	at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588)
> 	at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1826)
> 	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)
> 	at org.apache.spark.SparkContext.stop(SparkContext.scala:1825)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:587)
> 	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
> 	at scala.Option.getOrElse(Option.scala:121)
> 	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.executeSparkJob(AbstractCommonSparkTask.scala:91)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.runSparkJob(AbstractCommonSparkTask.scala:25)
> 	at com.asura.grinder.datatask.main.TaskMain$.main(TaskMain.scala:61)
> 	at com.asura.grinder.datatask.main.TaskMain.main(TaskMain.scala)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
> 	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> 17/06/20 17:24:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint
stopped!
> 17/06/20 17:24:37 INFO MemoryStore: MemoryStore cleared
> 17/06/20 17:24:37 INFO BlockManager: BlockManager stopped
> 17/06/20 17:24:37 INFO BlockManagerMaster: BlockManagerMaster stopped
> 17/06/20 17:24:37 WARN MetricsSystem: Stopping a MetricsSystem that is not running
> 17/06/20 17:24:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator
stopped!
> 17/06/20 17:24:37 INFO SparkContext: Successfully stopped SparkContext
> 17/06/20 17:24:37 ERROR MongoPilotTask: error occurred group{2}:task(222)
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at java.net.Socket.connect(Socket.java:538)
> 	at java.net.Socket.<init>(Socket.java:434)
> 	at java.net.Socket.<init>(Socket.java:244)
> 	at org.apache.spark.launcher.LauncherBackend.connect(LauncherBackend.scala:43)
> 	at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.start(StandaloneSchedulerBackend.scala:60)
> 	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
> 	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
> 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
> 	at scala.Option.getOrElse(Option.scala:121)
> 	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.executeSparkJob(AbstractCommonSparkTask.scala:91)
> 	at com.asura.grinder.datatask.task.AbstractCommonSparkTask.runSparkJob(AbstractCommonSparkTask.scala:25)
> 	at com.asura.grinder.datatask.main.TaskMain$.main(TaskMain.scala:61)
> 	at com.asura.grinder.datatask.main.TaskMain.main(TaskMain.scala)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
> 	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message