spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Leith (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings
Date Tue, 06 Oct 2015 08:24:26 GMT
Ewan Leith created SPARK-10947:
----------------------------------

             Summary: With schema inference from JSON into a Dataframe, add option to infer
all primitive object types as strings
                 Key: SPARK-10947
                 URL: https://issues.apache.org/jira/browse/SPARK-10947
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.5.1
            Reporter: Ewan Leith
            Priority: Minor


Currently, when a schema is inferred from a JSON file using sqlContext.read.json, the primitive
object types are inferred as string, long, boolean, etc.

However, if the inferred type is too specific (JSON obviously does not enforce types itself),
this causes issues with merging dataframe schemas.

Instead, we would like an option in the JSON inferField function to treat all primitive objects
as strings.

We'll create and submit a pull request for this for review.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message