spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akopich <...@git.apache.org>
Subject [GitHub] spark pull request #19565: [SPARK-22111][MLLIB] OnlineLDAOptimizer should fi...
Date Thu, 26 Oct 2017 18:36:21 GMT
Github user akopich commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19565#discussion_r147230366
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala ---
    @@ -446,14 +445,14 @@ final class OnlineLDAOptimizer extends LDAOptimizer with Logging
{
       override private[clustering] def next(): OnlineLDAOptimizer = {
         val batch = docs.sample(withReplacement = sampleWithReplacement, miniBatchFraction,
           randomGenerator.nextLong())
    -    if (batch.isEmpty()) return this
    --- End diff --
    
    But performance wise... Currently we have a single pass. But if we move the check to `next`,
`isEmpty` being an action will trigger the materialization of `batch` and then `batch` will
be materialized once again when `treeAggregate` is called in `submitMiniBatch`. 
    
    Right?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message