spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-20531) Spark master shouldn't send its address back to the workers over proxied connections
Date Sat, 29 Apr 2017 01:16:04 GMT
Sameer Agarwal created SPARK-20531:
--------------------------------------

             Summary: Spark master shouldn't send its address back to the workers over proxied
connections
                 Key: SPARK-20531
                 URL: https://issues.apache.org/jira/browse/SPARK-20531
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.0.2, 1.6.3, 2.1.2, 2.2.0
            Reporter: Sameer Agarwal


Currently, when a Spark worker connects to Spark master, the master sends its address back
to the worker (as part of the {{RegisteredWorker}} message). The worker then saves this address
and use it to talk to the master. The reason behind this handshake protocol is that if the
master goes down, a new master can always send back {{RegisteredWorker}} messages to all the
workers with its (new) IP address.

However, if there is a proxy between the master and worker, this unfortunately ends up bypassing
the proxy. A simple fix here is that we should encode the  "destination address" in the {{RegisterWorker}}
that can then be sent back to the to the worker as part of the {{RegisteredWorker}} message.

cc [~zsxwing]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message