spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] maropu commented on a change in pull request #29587: [SPARK-32376][SQL] Make unionByName null-filling behavior work with struct columns
Date Wed, 07 Oct 2020 14:00:38 GMT

maropu commented on a change in pull request #29587:
URL: https://github.com/apache/spark/pull/29587#discussion_r501036503



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2721,6 +2721,16 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val UNION_BYNAME_STRUCT_SUPPORT_ENABLED =
+    buildConf("spark.sql.unionByName.structSupport.enabled")
+      .doc("When true, the `allowMissingColumns` feature of `Dataset.unionByName` supports
" +
+        "nested column in struct types. Missing nested columns of struct columns with same
" +
+        "name will also be filled with null values. This currently does not support nested
" +
+        "columns in array and map types.")

Review comment:
       Thanks for the update. Btw, to follow a top-level case (See an example below), is it
technically difficult to reorder struct fields by adding `Project` on the top of `Union`?
Does this make the logic ugly?
   ```
   // The top-level case
   scala> val df1 = spark.range(1).selectExpr("id c", "id a")
   scala> val df2 = spark.range(1).selectExpr("id c", "id b")
   scala> df1.unionByName(df2, true).printSchema
   root
    |-- c: long (nullable = false)
    |-- a: long (nullable = true)
    |-- b: long (nullable = true)
   
   // The struct case (the current behaviour)
   scala> val df3 = spark.range(1).selectExpr("named_struct('c', id, 'a', id) col")
   scala> val df4 = spark.range(1).selectExpr("named_struct('c', id, 'b', id) col")
   scala> df3.unionByName(df4, true).printSchema
   root
    |-- col: struct (nullable = false)
    |    |-- a: long (nullable = true)
    |    |-- b: long (nullable = true)
    |    |-- c: long (nullable = false)
   
   // The following column order is more natural?
   scala> df3.unionByName(df4, true).printSchema
   root
    |-- col: struct (nullable = false)
    |    |-- c: long (nullable = true)
    |    |-- a: long (nullable = true)
    |    |-- b: long (nullable = false)
   
   == Analyzed Logical Plan ==
   col: struct<c:bigint,a:bigint,b:bigint>
   Project [struct(c, col#114.c, a, col#114.a, b, col#114.b) AS col#116] <=== Adding `Project`
in the end of `ResolveUnion`?
   +- Union false, false
      :- Project [named_struct(a, col#87.a, b, null, c, col#87.c) AS col#114]
      :  +- Project [named_struct(c, id#85L, a, id#85L) AS col#87]
      :     +- Range (0, 1, step=1, splits=Some(4))
      +- Project [named_struct(a, null, b, col#91.b, c, col#91.c) AS col#113]
         +- Project [named_struct(c, id#89L, b, id#89L) AS col#91]
            +- Range (0, 1, step=1, splits=Some(4))
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message