spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Lande <>
Subject Re: Spark SQL JSON Column Support
Date Wed, 28 Sep 2016 18:02:45 GMT
We are currently pulling out the JSON columns, passing them through
read.json, and then joining them back onto the initial DF so something like
from_json would be a nice quality of life improvement for us.

On Wed, Sep 28, 2016 at 10:52 AM, Michael Armbrust <>

> Spark SQL has great support for reading text files that contain JSON data.
> However, in many cases the JSON data is just one column amongst others.
> This is particularly true when reading from sources such as Kafka. This PR
> <> adds a new functions
> from_json that converts a string column into a nested StructType with a
> user specified schema, using the same internal logic as the json Data
> Source.
> Would love to hear any comments / suggestions.
> Michael

View raw message