spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] dongjoon-hyun commented on a change in pull request #27456: [SPARK-25040][SQL][FOLLOWUP] Add legacy config for allowing empty strings for certain types in json parser
Date Tue, 04 Feb 2020 20:57:34 GMT
dongjoon-hyun commented on a change in pull request #27456: [SPARK-25040][SQL][FOLLOWUP] Add
legacy config for allowing empty strings for certain types in json parser
URL: https://github.com/apache/spark/pull/27456#discussion_r374918341
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -37,7 +37,7 @@ license: |
 
   - Since Spark 3.0, the Dataset and DataFrame API `unionAll` is not deprecated any more.
It is an alias for `union`.
 
-  - In Spark version 2.4 and earlier, the parser of JSON data source treats empty strings
as null for some data types such as `IntegerType`. For `FloatType` and `DoubleType`, it fails
on empty strings and throws exceptions. Since Spark 3.0, we disallow empty strings and will
throw exceptions for data types except for `StringType` and `BinaryType`.
+  - In Spark version 2.4 and earlier, the parser of JSON data source treats empty strings
as null for some data types such as `IntegerType`. For `FloatType`, `DoubleType`, `DateType`
and `TimestampType`, it fails on empty strings and throws exceptions. Since Spark 3.0, we
disallow empty strings and will throw exceptions for data types except for `StringType` and
`BinaryType`. The previous behaviour of allowing empty string can be restored by setting `spark.sql.legacy.json.allowEmptyString.enabled`
to `true`.
 
 Review comment:
   Thank you for updating this, @viirya .

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message