spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-6331) New Spark Master URL is not picked up when streaming context is started from checkpoint
Date Fri, 13 Mar 2015 21:56:38 GMT
Tathagata Das created SPARK-6331:
------------------------------------

             Summary: New Spark Master URL is not picked up when streaming context is started
from checkpoint
                 Key: SPARK-6331
                 URL: https://issues.apache.org/jira/browse/SPARK-6331
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.2.1, 1.1.1, 1.3.0
            Reporter: Tathagata Das
            Assignee: Tathagata Das


When the SparkConf is reconstructed based on the checkpointed configuration, it recovers the
old master URL. This okay if the cluster on which the streaming application is relaunched
is the same cluster as it was running before. But if that cluster changes, there is no way
to inject the new master URL of the new cluster. As a result, the restarted app tries to connect
to the non-existent old cluster and fails. 

The solution is to check whether a master URL is set in the System properties (by Spark submit)
before recreating the SparkConf. If a new master url is set in the properties, then use it
as that is obviously the most relevant one. Otherwise load the old one (to maintain existing
behavior). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message