spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tdas <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-6752][Streaming] Allow StreamingContext...
Date Wed, 22 Apr 2015 07:41:44 GMT
Github user tdas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5428#discussion_r28850515
  
    --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
    @@ -621,19 +636,59 @@ object StreamingContext extends Logging {
           hadoopConf: Configuration = new Configuration(),
           createOnError: Boolean = false
         ): StreamingContext = {
    -    val checkpointOption = try {
    -      CheckpointReader.read(checkpointPath,  new SparkConf(), hadoopConf)
    -    } catch {
    -      case e: Exception =>
    -        if (createOnError) {
    -          None
    -        } else {
    -          throw e
    -        }
    -    }
    +    val checkpointOption = CheckpointReader.read(
    +      checkpointPath, new SparkConf(), hadoopConf, createOnError)
         checkpointOption.map(new StreamingContext(null, _, null)).getOrElse(creatingFunc())
       }
     
    +
    +  /**
    +   * Either recreate a StreamingContext from checkpoint data or create a new StreamingContext.
    +   * If checkpoint data exists in the provided `checkpointPath`, then StreamingContext
will be
    +   * recreated from the checkpoint data. If the data does not exist, then the StreamingContext
    +   * will be created by called the provided `creatingFunc` on the provided `sparkContext`.
Note
    +   * that the SparkConf configuration in the checkpoint data will not be restored as
the
    +   * SparkContext has already been created.
    +   *
    +   * @param checkpointPath Checkpoint directory used in an earlier StreamingContext program
    +   * @param creatingFunc   Function to create a new StreamingContext using the given
SparkContext
    +   * @param sparkContext   SparkContext using which the StreamingContext will be created
    +   */
    +  def getOrCreate(
    --- End diff --
    
    This is needed because there is already a version of getOrCreate with default arguments,
and there can be only one overloaded version with default args. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message