spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-22201) Dataframe describe includes string columns
Date Thu, 05 Oct 2017 00:05:00 GMT


Sean Owen commented on SPARK-22201:

If you're saying that string-valued columns should never appear in the output: not sure I
agree there's a problem. pandas will also output stats for such columns, for example, and
simply leave mean/stdev blank or NaN. The behavior exists now, so would have to be a good
reason that it's not allowed at all. 

If you're just saying it shouldn't be there by default, again not sure that's the least surprising
thing. The API lets you choose a subset of columns. The natural default with no args is to
show all cols.

pandas seems to have an argument that controls whether to show string cols or not. That's
possible here too but so is simply selecting the columns of interest.

> Dataframe describe includes string columns
> ------------------------------------------
>                 Key: SPARK-22201
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: cold gin
> As per the api documentation, the default no-arg Dataframe describe() function should
only include numerical column types, but it is including String types as well. This creates
unusable statistical results (for example, max returns "V8903" for one of the string columns
in my dataset), and this also leads to stacktraces when you run show() on the resulting dataframe
returned from describe().
> There also appears to be several related issues to this:
> But SPARK-16429 does not make sense with what the default api says, and only Int, Double,
etc (numeric) columns should be included when generating the statistics. 
> Perhaps this reveals the need for a new function to produce stats that make sense only
for string columns, or else an additional parameter to describe() to filter in/out certain
column types? 
> In summary, the *default* describe api behavior (no arg behavior) should not include
string columns. Note that boolean columns are correctly excluded by describe()

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message