spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] AngersZhuuuu commented on a change in pull request #33363: [SPARK-36156][SQL] SCRIPT TRANSFORM ROW FORMAT DELIMITED should respect `NULL DEFINED AS` and default value should be `\N`
Date Thu, 22 Jul 2021 03:05:50 GMT

AngersZhuuuu commented on a change in pull request #33363:
URL: https://github.com/apache/spark/pull/33363#discussion_r674468765



##########
File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql
##########
@@ -121,6 +121,38 @@ USING 'cat' AS (d)
   NULL DEFINED AS 'NULL'
 FROM t;
 
+SELECT TRANSFORM(a, b, c, null)
+  ROW FORMAT DELIMITED
+  FIELDS TERMINATED BY '@'
+  LINES TERMINATED BY '\n'
+USING 'cat' AS (a, b, c, d)
+  ROW FORMAT DELIMITED
+  FIELDS TERMINATED BY '@'
+  LINES TERMINATED BY '\n'
+  NULL DEFINED AS 'NULL'

Review comment:
       > Can you explain it instead of just throwing out a screenshot? What's the behavior
of Hive if `NULL DEFINED AS '\n'`?
   
   Sorry for miss this.
   
   If hive set `NULL DEFINED AS '\n'`, null value will be converted to `\n` and when handle
output, we split  line with `\n` first.  then the string `\n` for replacing null value was
regarded as line separator too and split a new line.
   Such as we have one line inpurt
   ```
   1@true@Spark SQL@\n
   ```
   
   then it will split to two line output
   ```
   1@true@Spark SQL@
   
   ```
   the first line will be split as "1", "true", "Spark SQL", ""
   the second line was split as ""
   After pad null value.the result is 
   ```
   1   true        Spark SQL
        NULL     NULL              NULL
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message