Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 15 Mar 2017 00:04:41 +0000 (UTC)
From: "Hyukjin Kwon (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13055979.1489507928000.20969.1489536281638@Atlassian.JIRA>
In-Reply-To: <JIRA.13055979.1489507928000@Atlassian.JIRA>
References: <JIRA.13055979.1489507928000@Atlassian.JIRA> <JIRA.13055979.1489507928937@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-19950) nullable ignored when df.load() is
 executed for file-based data source
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 15 Mar 2017 00:04:48 -0000


    [ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925301#comment-15925301 ] 

Hyukjin Kwon commented on SPARK-19950:
--------------------------------------

[~kiszk], do you think we maybe resolve this JIRA as a duplicate of SPARK-16472 if this describes file format datasources force the schema into nullable ones?

> nullable ignored when df.load() is executed for file-based data source
> ----------------------------------------------------------------------
>
>                 Key: SPARK-19950
>                 URL: https://issues.apache.org/jira/browse/SPARK-19950
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Kazuaki Ishizaki
>
> This problem is reported in [Databricks forum|https://forums.databricks.com/questions/7123/nullable-seemingly-ignored-when-reading-parquet.html].
> When we execute the following code, a schema for "id" in {{dfRead}} has {{nullable = true}}. It should be {{nullable = false}}.
> {code:java}
> val field = "id"
> val df = spark.range(0, 5, 1, 1).toDF(field)
> val fmt = "parquet"
> val path = "/tmp/parquet"
> val schema = StructType(Seq(StructField(field, LongType, false)))
> df.write.format(fmt).mode("overwrite").save(path)
> val dfRead = spark.read.format(fmt).schema(schema).load(path)
> dfRead.printSchema
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org