spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navya Krishnappa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20152) Time zone is not respected while parsing csv for timeStampFormat "MM-dd-yyyy'T'HH:mm:ss.SSSZZ"
Date Thu, 30 Mar 2017 12:48:42 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948989#comment-15948989
] 

Navya Krishnappa commented on SPARK-20152:
------------------------------------------

According to the spark "yyyy-MM-dd'T'HH:mm:ss.SSSZZ" is default timestamp format. In examples,
i have swapped the date fields. And I'm using valid letters in my format.

> Time zone is not respected while parsing csv for timeStampFormat "MM-dd-yyyy'T'HH:mm:ss.SSSZZ"
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-20152
>                 URL: https://issues.apache.org/jira/browse/SPARK-20152
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Navya Krishnappa
>
> When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ",
time zone is ignored.
> Source File: 
> TimeColumn
> 03-21-2017T03:30:02Z
> Source code1:
> Dataset dataset = getSqlContext().read()
> .option(PARSER_LIB, "commons")
> .option(INFER_SCHEMA, "true")
> .option(DELIMITER, ",")
> .option(QUOTE, "\"")
> .option(ESCAPE, "\\")
> .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ")
> .option(MODE, Mode.PERMISSIVE)
> .csv(sourceFile);
> Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but expected result
is TimeCoumn should be of "TimestampType"  and should consider time zone for manipulation
> Source code2: 
> Dataset dataset = getSqlContext().read() 
> .option(PARSER_LIB, "commons") 
> .option(INFER_SCHEMA, "true") 
> .option(DELIMITER, ",") 
> .option(QUOTE, "\"") 
> .option(ESCAPE, "\\") 
> .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss") 
> .option(MODE, Mode.PERMISSIVE) 
> .csv(sourceFile); 
> Result: TimeColumn [ TimestampType] and value is "2017-03-21 03:30:02.0", but expected
result is TimeCoumn should consider time zone for manipulation



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message