spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Teng Peng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24431) wrong areaUnderPR calculation in BinaryClassificationEvaluator
Date Sat, 02 Jun 2018 15:40:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499109#comment-16499109
] 

Teng Peng commented on SPARK-24431:
-----------------------------------

I am trying to understand this description. What's your definition of event rate? Is it defined
as (TP+FN)/(TP + FP + TN + FN).

> wrong areaUnderPR calculation in BinaryClassificationEvaluator 
> ---------------------------------------------------------------
>
>                 Key: SPARK-24431
>                 URL: https://issues.apache.org/jira/browse/SPARK-24431
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>    Affects Versions: 2.2.0
>            Reporter: Xinyong Tian
>            Priority: Major
>
> My problem, I am using CrossValidator(estimator=LogisticRegression(...), ...,  evaluator=BinaryClassificationEvaluator(metricName='areaUnderPR')) 
to select best model. when the regParam in logistict regression is very high, no variable
will be selected (no model), ie every row 's prediction is same ,eg. equal event rate (baseline
frequency). But at this point,  BinaryClassificationEvaluator set the areaUnderPR highest.
As a result  best model seleted is a no model. 
> the reason is following.  at time of no model, precision recall curve will be only two
points: at recall =0, precision should be set to  zero , while the software set it to 1.
at recall=1, precision is the event rate. As a result, the areaUnderPR will be close 0.5
(my even rate is very low), which is maximum .
> the solution is to set precision =0 when recall =0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message