spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-24027) Support MapType(StringType, DataType) as root type by from_json
Date Thu, 19 Apr 2018 20:19:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-24027:
------------------------------------

    Assignee: Apache Spark

> Support MapType(StringType, DataType) as root type by from_json
> ---------------------------------------------------------------
>
>                 Key: SPARK-24027
>                 URL: https://issues.apache.org/jira/browse/SPARK-24027
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Maxim Gekk
>            Assignee: Apache Spark
>            Priority: Major
>
> Currently, *MapType* is not supported by the *from_json* function as the root type. For
example, the following code doesn't work on Spark 2.3:
> {code}
> scala> import org.apache.spark.sql.types._
> import org.apache.spark.sql.types._
> scala> val schema = MapType(StringType, IntegerType)
> schema: org.apache.spark.sql.types.MapType = MapType(StringType,IntegerType,true)
> scala> val in = Seq("""{"a": 1, "b": 2, "c": 3}""").toDS()
> in: org.apache.spark.sql.Dataset[String] = [value: string]
> scala> in.select(from_json($"value", schema, Map[String, String]())).collect()
> org.apache.spark.sql.AnalysisException: cannot resolve 'jsontostructs(`value`)' due to
data type mismatch: Input schema map<string,int> must be a struct or an array of structs.
> {code}
> Purpose of the ticket is to support _MapType with StringType as keys type_.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message