spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gatorsmile <...@git.apache.org>
Subject [GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestam...
Date Fri, 20 Oct 2017 08:16:45 GMT
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18664#discussion_r145902248
  
    --- Diff: python/pyspark/sql/types.py ---
    @@ -1619,11 +1619,39 @@ def to_arrow_type(dt):
             arrow_type = pa.decimal(dt.precision, dt.scale)
         elif type(dt) == StringType:
             arrow_type = pa.string()
    +    elif type(dt) == DateType:
    +        arrow_type = pa.date32()
    +    elif type(dt) == TimestampType:
    +        # Timestamps should be in UTC, JVM Arrow timestamps require a timezone to be
read
    +        arrow_type = pa.timestamp('us', tz='UTC')
         else:
             raise TypeError("Unsupported type in conversion to Arrow: " + str(dt))
         return arrow_type
     
     
    +def _check_dataframe_localize_timestamps(df):
    +    """ Convert timezone aware timestamps to timezone-naive in local time
    +    """
    +    from pandas.api.types import is_datetime64tz_dtype
    +    for column, series in df.iteritems():
    +        # TODO: handle nested timestamps?
    +        if is_datetime64tz_dtype(series.dtype):
    +            df[column] = series.dt.tz_convert('tzlocal()').dt.tz_localize(None)
    +    return df
    +
    +
    +def _check_series_convert_timestamps_internal(s):
    +    """ Convert a tz-naive timestamp in local tz to UTC normalized for Spark internal
storage
    +    """
    +    from pandas.api.types import is_datetime64_dtype
    +    # TODO: handle nested timestamps?
    --- End diff --
    
    What is the behavior for nested timestamps? Give an example?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message