spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ueshin <...@git.apache.org>
Subject [GitHub] spark pull request #21704: [SPARK-24734][SQL] Fix containsNull of Concat for...
Date Wed, 04 Jul 2018 09:15:52 GMT
Github user ueshin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21704#discussion_r200060536
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
    @@ -2007,7 +2007,14 @@ case class Concat(children: Seq[Expression]) extends Expression
{
         }
       }
     
    -  override def dataType: DataType = children.map(_.dataType).headOption.getOrElse(StringType)
    +  override def dataType: DataType = {
    +    val dataTypes = children.map(_.dataType)
    +    dataTypes.headOption.map {
    +      case ArrayType(et, _) =>
    +        ArrayType(et, dataTypes.exists(_.asInstanceOf[ArrayType].containsNull))
    +      case dt => dt
    +    }.getOrElse(StringType)
    +  }
    --- End diff --
    
    Yeah, `Coalesce` should be fixed, and `Least` and `Greatest` are also suspicious.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message