spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MLnick <...@git.apache.org>
Subject [GitHub] spark issue #18554: [SPARK-21306][ML] OneVsRest should cache weightCol if ne...
Date Tue, 11 Jul 2017 15:00:30 GMT
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/18554
  
    Oh, of course some classifiers may not have weight col. in which case it
    will be ignored - though we should probably log a warning in this case.
    On Tue, 11 Jul 2017 at 16:58, Nick Pentreath <nick.pentreath@gmail.com>
    wrote:
    
    > I agree on this
    > On Tue, 11 Jul 2017 at 16:53, Yanbo Liang <notifications@github.com>
    > wrote:
    >
    >> *@yanboliang* commented on this pull request.
    >> ------------------------------
    >>
    >> In
    >> mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
    >> <https://github.com/apache/spark/pull/18554#discussion_r126710934>:
    >>
    >> > @@ -317,7 +318,12 @@ final class OneVsRest @Since("1.4.0") (
    >>      val numClasses = MetadataUtils.getNumClasses(labelSchema).fold(computeNumClasses())(identity)
    >>      instr.logNumClasses(numClasses)
    >>
    >> -    val multiclassLabeled = dataset.select($(labelCol), $(featuresCol))
    >> +    val multiclassLabeled = getClassifier match {
    >> +      // SPARK-21306: cache weightCol if necessary
    >> +      case c: HasWeightCol if c.isDefined(c.weightCol) && c.getWeightCol.nonEmpty
=>
    >> +        dataset.select($(labelCol), $(featuresCol), c.getWeightCol)
    >> +      case _ => dataset.select($(labelCol), $(featuresCol))
    >> +    }
    >>
    >> OneVsRest is a classification estimator, I think we should make weightCol
    >> a member param of it like featuresCol. For example:
    >>
    >> val dataset = dataset       // This dataset has column: a, b, c.
    >> val ova = new OneVsRest().setFeaturesCol("a").setClassifier(new LogisticRegression().setFeaturesCol("b"))
    >>
    >> The features column used by OneVsRest is a. The features column set for
    >> OneVsRest will override corresponding set in OneVsRest.classifier. So we
    >> should follow this way for weightCol as well. Thanks.
    >>
    >> —
    >> You are receiving this because you commented.
    >> Reply to this email directly, view it on GitHub
    >> <https://github.com/apache/spark/pull/18554#pullrequestreview-49220188>,
    >> or mute the thread
    >> <https://github.com/notifications/unsubscribe-auth/AA_SBzWk_xBWbVhuw3Q60BJdQnvPlKnAks5sM4xsgaJpZM4OPag0>
    >> .
    >>
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message