spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwws <...@git.apache.org>
Subject [GitHub] spark pull request: [Spark-13374][Streaming][wip] make it possible...
Date Fri, 19 Feb 2016 01:19:07 GMT
Github user mwws commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11249#discussion_r53410683
  
    --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
    @@ -269,6 +270,39 @@ class StreamingContext private[streaming] (
         RDDOperationScope.withScope(sc, name, allowNesting = false, ignoreParent = false)(body)
       }
     
    +  private[streaming] val recoverableAccuNameToId: mutable.Map[String, Long] = {
    +    if (isCheckpointPresent) {
    +      // accumulators created by StreamingContext must provide name, so it's safe to
call name.get
    +      mutable.Map(_cp.trackedAccs.map(accCP => (accCP.name, -1L)).toSeq: _*)
    +    } else {
    +      mutable.Map.empty
    +    }
    +  }
    +
    +  /**
    +    * Different from accumulator in SparkContext, it will first try to recover from Checkpoint
    +    * if it exist.
    +    *
    +    * @param initialValue   initial value of accumulator. It will be ignored when recovering
from
    +    *                       checkpoint
    +    * @param name           name is required as identity to find corresponding accumulator.
    +    */
    +  def accumulator[T](initialValue: T, name: String)(implicit param: AccumulatorParam[T])
    --- End diff --
    
    hmm... change func name might me a good idea to explicitly emphasize the recoverable feature.
I will change it. 
    
    About the input parameter, I don't think `createFunc `here is necessary, `initialValue`
should be enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message