spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yuhao yang (JIRA)" <>
Subject [jira] [Resolved] (SPARK-12875) Add Weight of Evidence and Information value to as a feature transformer
Date Mon, 24 Jul 2017 18:29:00 GMT


yuhao yang resolved SPARK-12875.
    Resolution: Won't Do

> Add Weight of Evidence and Information value to as a feature transformer
> ---------------------------------------------------------------------------------
>                 Key: SPARK-12875
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: ML
>            Reporter: yuhao yang
>            Priority: Minor
> As a feature transformer, WOE and IV enable one to:
> Consider each variable’s independent contribution to the outcome.
> Detect linear and non-linear relationships.
> Rank variables in terms of "univariate" predictive strength.
> Visualize the correlations between the predictive variables and the binary outcome.
> gives a good introduction
to WoE and IV.
>  The Weight of Evidence or WoE value provides a measure of how well a grouping of feature
is able to distinguish between a binary response (e.g. "good" versus "bad"), which is widely
used in grouping continuous feature or mapping categorical features to continuous values.
It is computed from the basic odds ratio:
> (Distribution of positive Outcomes) / (Distribution of negative Outcomes)
> where Distr refers to the proportion of positive or negative in the respective group,
relative to the column totals.
> The WoE recoding of features is particularly well suited for subsequent modeling using
Logistic Regression or MLP.
> In addition, the information value or IV can be computed based on WoE, which is a popular
technique to select variables in a predictive model.
> TODO: Currently we support only calculation for categorical features. Add an estimator
to estimate the proper grouping for continuous feature. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message