spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From skparkes <>
Subject [GitHub] spark pull request: [SPARK-13842][PYSPARK] pyspark.sql.types.Struc...
Date Fri, 08 Apr 2016 02:12:38 GMT
Github user skparkes commented on a diff in the pull request:
    --- Diff: python/pyspark/sql/ ---
    @@ -507,20 +503,52 @@ def add(self, field, data_type=None, nullable=True, metadata=None):
                     data_type_f = data_type
                 self.fields.append(StructField(field, data_type_f, nullable, metadata))
    -            self.names.append(field)
    -        self._needSerializeAnyField = any(f.needConversion() for f in self.fields)
             return self
    +    @property
    +    def names(self):
    +        """Return a list of the field names"""
    +        return [ for field in self]
    +    @property
    +    def _needSerializeAnyField(self):
    +        """Determine if any field needs conversion"""
    +        return any(field.needConversion() for field in self)
    +    def __iter__(self):
    +        """Iterate the fields"""
    +        return iter(self.fields)
    +    def __len__(self):
    +        """Return the number of fields."""
    +        return len(self.fields)
    +    def __getitem__(self, key):
    +        """Access fields by name or slice."""
    +        if isinstance(key, str):
    +            _dict_fields = { field for field in self}
    --- End diff --
    I figured the performance of the metadata manipulation would be negligible as soon as
you interacted with even a small dataset.
    I could look into other lookup methods, but this seemed the most straightforward and pythonic.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message