spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srowen <...@git.apache.org>
Subject [GitHub] spark pull request #22784: [SPARK-25790][MLLIB] PCA: Support more than 65535...
Date Sat, 27 Oct 2018 19:57:49 GMT
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22784#discussion_r228724515
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/PCA.scala ---
    @@ -49,7 +50,16 @@ class PCA @Since("1.4.0") (@Since("1.4.0") val k: Int) {
           "Try reducing the parameter k for PCA, or reduce the input feature " +
           "vector dimension to make this tractable.")
     
    -    val mat = new RowMatrix(sources)
    +    val mat = if (numFeatures > 65535) {
    +      val meanVector = Statistics.colStats(sources).mean
    --- End diff --
    
    Rather than call `.toArray` and `.zipped` below, can this not be written as Vector - Vector
in the loop below? might be more efficient.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message