spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric O. LEBIGOT (EOL) (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-5820) Example does not work when using SOCKS proxy
Date Sun, 15 Feb 2015 15:29:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322022#comment-14322022
] 

Eric O. LEBIGOT (EOL) edited comment on SPARK-5820 at 2/15/15 3:28 PM:
-----------------------------------------------------------------------

Thank you for investigating this.

The proxy server (localhost) does not require authentication.

The example (SparkPi) does work when no SOCKS proxy is used (in the OS X Network Preferences).

I did not mean to say that my proxy doesn't work for any JVM-based process (I'm not sure how
I could test this). :) If you were referring to the three JVM settings that I tried (with
the local SOCKS proxy server active), they are simply those of the OS X Java Preferences
(Apple>System Preferences>Java>Network Settings). For all JVM proxy settings, ./bin/run-example
SparkPi fails in the same way (as in the original report).

Here is how to reproduce the problem:
1) Creation of the local SOCKS proxy: ssh -g -N -D 12345 login@remotehost
2) Activation of the proxy in OS X: Apple>System Preferences>Network>Advanced>Proxies,
select SOCKS Proxy, set SOCKS Proxy Server to localhost, port 12345, make sure that Bypassed
hosts contains localhost (and 127.0.0.1, for good measure, maybe).

Now, connections go through the proxy (this should show in http://www.whatismyip.com/, for
example)… and the SparkPi example fails.


was (Author: lebigot):
Thank you for investigating this.

The proxy server (localhost) does not require authentication.

The example (SparkPi) does work when no SOCKS proxy is used (in the OS X Network Preferences).

I did not mean to say that my proxy doesn't work for any JVM-based process (I'm not sure how
I could test this). :) If you were referring to the three JVM settings that I tried (with
the local SOCKS proxy server active), they are simply those of the OS X Java Preferences
(Apple>System Preferences>Java>Network Settings). For all JVM proxy settings, ./bin/run-example
SparkPi fails in the same way (as in the original report).

Here is how to reproduce the problem:
1) Creation of the local SOCKS proxy: ssh -g -N -D 12345 login@remotehost
2) Activation of the proxy in OS X: Apple>System Preferences>Network>Advanced>Proxies,
select SOCKS Proxy, set SOCKS Proxy Server to localhost, port 12345, make sure that Bypassed
hosts contains localhost (and 127.0.0.1, for good measure, maybe).

Now, connections go through the proxy (this should show in http://www.whatismyip.com/, for
example).

> Example does not work when using SOCKS proxy
> --------------------------------------------
>
>                 Key: SPARK-5820
>                 URL: https://issues.apache.org/jira/browse/SPARK-5820
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 1.2.1
>            Reporter: Eric O. LEBIGOT (EOL)
>
> When using a SOCKS proxy (on OS X 10.10.2), running even the basic example ./bin/run-example
SparkPi 10 fails.
> -- Partial log --
> 15/02/14 23:23:00 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting
job
> 15/02/14 23:23:00 INFO TaskSchedulerImpl: Cancelling stage 0
> 15/02/14 23:23:00 INFO Executor: Executor is trying to kill task 1.0 in stage 0.0 (TID
1)
> 15/02/14 23:23:00 INFO TaskSchedulerImpl: Stage 0 was cancelled
> 15/02/14 23:23:00 INFO DAGScheduler: Job 0 failed: reduce at SparkPi.scala:35, took 1.920223
s
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage
failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0
(TID 0, localhost): java.net.SocketException: Malformed reply from SOCKS server
>         at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:129)
>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:503)
>         at java.net.Socket.connect(Socket.java:579)
>         at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>         at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>         at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>         at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>         at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>         at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1003)
>         at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:951)
>         at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
>         at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:582)
>         at org.apache.spark.util.Utils$.fetchFile(Utils.scala:433)
>         at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:356)
>         at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:353)
>         at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>         at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>         at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
>         at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
>         at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)
>         at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:353)
>         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:181)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> Driver stacktrace:
>         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
>         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
>         at scala.Option.foreach(Option.scala:245)
>         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
>         at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>         at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>         at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>         at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message