spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Lande <nathanla...@gmail.com>
Subject Re: Spark SQL JSON Column Support
Date Wed, 28 Sep 2016 18:02:45 GMT
We are currently pulling out the JSON columns, passing them through
read.json, and then joining them back onto the initial DF so something like
from_json would be a nice quality of life improvement for us.

On Wed, Sep 28, 2016 at 10:52 AM, Michael Armbrust <michael@databricks.com>
wrote:

> Spark SQL has great support for reading text files that contain JSON data.
> However, in many cases the JSON data is just one column amongst others.
> This is particularly true when reading from sources such as Kafka. This PR
> <https://github.com/apache/spark/pull/15274> adds a new functions
> from_json that converts a string column into a nested StructType with a
> user specified schema, using the same internal logic as the json Data
> Source.
>
> Would love to hear any comments / suggestions.
>
> Michael
>

Mime
View raw message