spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maropu <...@git.apache.org>
Subject [GitHub] spark pull request #20757: [SPARK-23595][SQL] ValidateExternalType should su...
Date Thu, 08 Mar 2018 13:28:03 GMT
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20757#discussion_r173156309
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
    @@ -1416,13 +1417,43 @@ case class ValidateExternalType(child: Expression, expected: DataType)
     
       override def nullable: Boolean = child.nullable
     
    -  override def dataType: DataType = RowEncoder.externalDataTypeForInput(expected)
    -
    -  override def eval(input: InternalRow): Any =
    -    throw new UnsupportedOperationException("Only code-generated evaluation is supported")
    +  override val dataType: DataType = RowEncoder.externalDataTypeForInput(expected)
     
       private val errMsg = s" is not a valid external type for schema of ${expected.simpleString}"
     
    +
    +  private lazy val checkType: (Any) => Boolean = expected match {
    +    case _: DecimalType =>
    +      (value: Any) => {
    +        value.isInstanceOf[java.math.BigDecimal] || value.isInstanceOf[scala.math.BigDecimal]
||
    +          value.isInstanceOf[Decimal]
    +      }
    +    case _: ArrayType =>
    +      (value: Any) => {
    +        value.getClass.isArray || value.isInstanceOf[Seq[_]]
    +      }
    +    case _ =>
    +      val dataTypeClazz = if (dataType.isInstanceOf[ObjectType]) {
    --- End diff --
    
    Both types (`PythonUserDefinedType` or `UserDefinedType`) are converted into native types
or ObjectType in `externalDataTypeForInput`. So, IIUC we can handle the two types here, too.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message