spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MaxGekk <...@git.apache.org>
Subject [GitHub] spark pull request #21472: [SPARK-24445][SQL] Schema in json format for from...
Date Fri, 01 Jun 2018 10:38:32 GMT
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21472#discussion_r192358871
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
    @@ -747,8 +748,13 @@ case class StructsToJson(
     
     object JsonExprUtils {
     
    -  def validateSchemaLiteral(exp: Expression): StructType = exp match {
    -    case Literal(s, StringType) => CatalystSqlParser.parseTableSchema(s.toString)
    +  def validateSchemaLiteral(exp: Expression): DataType = exp match {
    +    case Literal(s, StringType) =>
    +      try {
    +        DataType.fromJson(s.toString)
    --- End diff --
    
    > Shall we add the support with type itself with CatalystSqlParser.parseDataType too?
    
    I will do but it won't solve customer's problem fully.
    
    > Also, are you able to use catalogString?
    
    I just check that:
    ```scala
    val schema = MapType(StringType, IntegerType).catalogString
    val ds = spark.sql(
          s"""
            |select from_json('{"a":1}', '$schema')
          """.stripMargin)
    ds.show()
    ```
    and got this one:
    
    ```
    extraneous input '<' expecting {'SELECT', 'FROM', ...}(line 1, pos 3)
    
    == SQL ==
    map<string,int>
    ---^^^
    ; line 2 pos 7
    ```
    
    The same with ` val schema = new StructType().add("a", IntegerType).catalogString
    `
    
    ```
    == SQL ==
    struct<a:int>
    ------^^^
    ; line 2 pos 7
    org.apache.spark.sql.AnalysisException
    ```
    
    Am I doing something wrong?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message