spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srowen <>
Subject [GitHub] spark pull request #14858: [SPARK-17219][ML] Add NaN value handling in Bucke...
Date Mon, 05 Sep 2016 08:58:42 GMT
Github user srowen commented on a diff in the pull request:
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala ---
    @@ -114,10 +115,10 @@ final class QuantileDiscretizer @Since("1.6.0") (@Since("1.6.0")
override val ui
         splits(0) = Double.NegativeInfinity
         splits(splits.length - 1) = Double.PositiveInfinity
    -    val distinctSplits = splits.distinct
    +    val distinctSplits = splits.filter(!_.isNaN).distinct
    --- End diff --
    What would you do with NaN in approxQuantile except ignore it? it can't affect a quantile.
The question of what the bucketizer should do with NaN is separate. I'd still favor one behavior
there, but keeping that separate, let's figure out what approxQuantile should do.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message