spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark
Date Thu, 04 May 2017 13:55:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996758#comment-15996758
] 

Hyukjin Kwon commented on SPARK-12467:
--------------------------------------

I actually quite like {{**kwargs}} usage and I think arguably it is straightforward and easier
than namedtuple like way. It is documented so if users are aware of this, probably, it is
not worth deprecating/removing yet.

We will anyway easily support this in the far future after dropping Python before 3.6.


> Get rid of sorting in Row's constructor in pyspark
> --------------------------------------------------
>
>                 Key: SPARK-12467
>                 URL: https://issues.apache.org/jira/browse/SPARK-12467
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 1.5.2
>            Reporter: Irakli Machabeli
>            Priority: Minor
>
> Current implementation of Row's __new__ sorts columns by name
> First of all there is no obvious reason to sort, second, if one converts dataframe to
rdd and than back to dataframe, order of column changes. While this is not  a bug, nevetheless
it makes looking at the data really inconvenient.
>     def __new__(self, *args, **kwargs):
>         if args and kwargs:
>             raise ValueError("Can not use both args "
>                              "and kwargs to create Row")
>         if args:
>             # create row class or objects
>             return tuple.__new__(self, args)
>         elif kwargs:
>             # create row objects
>             names = sorted(kwargs.keys()) # just get rid of sorting here!!!
>             row = tuple.__new__(self, [kwargs[n] for n in names])
>             row.__fields__ = names
>             return row
>         else:
>             raise ValueError("No args or kwargs")



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message