spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanbo Liang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-14657) RFormula output wrong features when formula w/o intercept
Date Fri, 15 Apr 2016 09:40:25 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242701#comment-15242701
] 

Yanbo Liang edited comment on SPARK-14657 at 4/15/16 9:39 AM:
--------------------------------------------------------------

More cases in R:

df=data.frame(income=c(5,5,3,3,6,5),
                       won=c(0,0,1,1,1,0),
                       age=c(18,18,23,50,19,39),
                       home=c(0,0,1,0,0,1))
{quote}
model <- glm(factor(won) ~ factor(income) + factor(age) + factor(home), data=df, family=binomial(link="logit"))

Coefficients: (3 not defined because of singularities)
                  Estimate Std. Error z value Pr(>|z|)
(Intercept)      2.457e+01  1.310e+05       0        1
factor(income)5 -4.913e+01  1.605e+05       0        1
factor(income)6 -2.573e-30  1.853e+05       0        1
factor(age)19           NA         NA      NA       NA
factor(age)23   -1.383e-30  1.853e+05       0        1
factor(age)39   -3.479e-14  1.605e+05       0        1
factor(age)50           NA         NA      NA       NA
factor(home)1           NA         NA      NA       NA
{quote}
model2 <- glm(factor(won) ~ factor(income) + factor(age) + factor(home) - 1, data=df, family=binomial(link="logit"))
model3 <- glm(factor(won) ~ factor(age) + factor(income) + factor(home) - 1, data=df, family=binomial(link="logit"))


was (Author: yanboliang):
More cases in R:
{quote}
df=data.frame(income=c(5,5,3,3,6,5),
                       won=c(0,0,1,1,1,0),
                       age=c(18,18,23,50,19,39),
                       home=c(0,0,1,0,0,1))
model <- glm(factor(won) ~ factor(income) + factor(age) + factor(home), data=df, family=binomial(link="logit"))
{quote}
Coefficients: (3 not defined because of singularities)
                  Estimate Std. Error z value Pr(>|z|)
(Intercept)      2.457e+01  1.310e+05       0        1
factor(income)5 -4.913e+01  1.605e+05       0        1
factor(income)6 -2.573e-30  1.853e+05       0        1
factor(age)19           NA         NA      NA       NA
factor(age)23   -1.383e-30  1.853e+05       0        1
factor(age)39   -3.479e-14  1.605e+05       0        1
factor(age)50           NA         NA      NA       NA
factor(home)1           NA         NA      NA       NA
{quote}
model2 <- glm(factor(won) ~ factor(income) + factor(age) + factor(home) - 1, data=df, family=binomial(link="logit"))
model3 <- glm(factor(won) ~ factor(age) + factor(income) + factor(home) - 1, data=df, family=binomial(link="logit"))

> RFormula output wrong features when formula w/o intercept
> ---------------------------------------------------------
>
>                 Key: SPARK-14657
>                 URL: https://issues.apache.org/jira/browse/SPARK-14657
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>            Reporter: Yanbo Liang
>
> SparkR::glm output different features compared with R glm.
> SparkR::glm
> {quote}
> training <- suppressWarnings(createDataFrame(sqlContext, iris))
> model <- glm(Sepal_Width ~ Sepal_Length + Species - 1, data = training)
> summary(model)
> Coefficients:
>                     Estimate  Std. Error  t value  Pr(>|t|)
> Sepal_Length        0.67468   0.0093013   72.536   0
> Species_versicolor  -1.2349   0.07269     -16.989  0
> Species_virginica   -1.4708   0.077397    -19.003  0
> {quote}
> stats::glm
> {quote}
> summary(glm(Sepal.Width ~ Sepal.Length + Species - 1, data = iris))
> Coefficients:
>                   Estimate Std. Error t value Pr(>|t|)    
> Sepal.Length        0.3499     0.0463   7.557 4.19e-12 ***
> Speciessetosa       1.6765     0.2354   7.123 4.46e-11 ***
> Speciesversicolor   0.6931     0.2779   2.494   0.0137 *  
> Speciesvirginica    0.6690     0.3078   2.174   0.0313 *  
> {quote}
> The encoder for string type feature is different. R did not drop any category but SparkR
drop the last one.
> I searched online and test some other cases, found when we fit R glm model(or other models
powered by R formula) w/o intercept on a dataset including string/category features, one of
the levels in the first category feature is being used as reference level, we will not drop
any category for that feature.
> I think we should keep consistent semantics between Spark RFormula and R formula.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message