spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-16467) After importing R data.frame, although DataFrame columns show . replaced by _, the describe() function gives warnings on . in the name
Date Mon, 11 Jul 2016 01:45:10 GMT

    [ https://issues.apache.org/jira/browse/SPARK-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370066#comment-15370066
] 

Dongjoon Hyun commented on SPARK-16467:
---------------------------------------

Thank YOU for reporting!

> After importing R data.frame, although DataFrame columns show . replaced by _,  the describe()
function gives warnings on . in the name
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16467
>                 URL: https://issues.apache.org/jira/browse/SPARK-16467
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>    Affects Versions: 1.6.1
>            Reporter: Neil Dewar
>            Priority: Minor
>
> When importing an R data.frame into a Spark DataFrame using createDataFrame(), if the
R data.frame column names contain period characters "." they are converted to underscores
"_".    Most SparkR functions then display the column names with the underscores.  
> If the describe() function is used, with the option for a specific function name, warning
messages are displayed about the use of period characters.
> Example:
> sdfIris <- createDataFrame(sqlContext, iris)
> str(sdfIris) # works fine
> collect(describe(sdfIris, Species))
> the last function above throws a string of warnings such as:
> 1: In FUN(X[[i]], ...) :  Use Sepal_Length instead of Sepal.Length  as column name
> Note, the describe() function appears to work ok if no column name is specified but fails
if a column name is specified, even if the specified name did not contain a period.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message