spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Baker (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-25943) Corruption when writing data into a catalog table with a different struct schema
Date Mon, 05 Nov 2018 10:43:02 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

James Baker updated SPARK-25943:
--------------------------------
    Description: 
Suppose I have a catalog table with schema StructType(Seq(StructField("a", StructType(Seq(StructField("b",
DataTypes.StringType), StructField("c", DataTypes.StringType))).

Suppose I now try to append a record to it:
{code:java}
{"a": {"c": "data1", "b": "data2"}}
{code}
That record will actually be appended as:
{code:java}
{"a": {"b": "data1", "c": "data2"}}
{code}
which is obviously not close to what the user wanted or expected (for me it silently corrupted
my data).

It turns out that the user could provide a totally different record,
{code:java}
{"a": {"this column": "is totally different", "but": "the types match up"}}
{code}
and it'd still get written out, but as
{code:java}
{"a": {"b": "is totally different", "c": "the types match up"}}
{code}
This is because [in DDLPreprocessingUtils.castAndRenameOutput|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500]
[,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500],]
Spark puts effort in to reordering column names in line with what the output expects, but
merely casts any other types. This works nicely in a case where you try to e.g. write an int
into a double field, but goes wrong on complex datatypes if the types match up but the field
names do not.

  was:
Suppose I have a catalog table with schema StructType(Seq(StructField("a", StructType(Seq(StructField("b",
DataTypes.StringType), StructField("c", DataTypes.StringType))).

Suppose I now try to append a record to it:
{code:java}
{"a": {"c": "data1", "b": "data2"}}
{code}
That record will actually be appended as:
{code:java}
{"a": {"b": "data1", "c": "data2"}}
{code}
which is obviously not close to what the user wanted or expected (for me it silently corrupted
my data).

It turns out that the user could provide a totally different record,
{code:java}
{"a": {"this column": "is totally different", "but": "the types match up"}}
{code}
and it'd still get written out, but as
{code:java}
{"a": {"b": "is totally different", "c": "the types match up"}}
{code}
This is because [in DDLPreprocessingUtils.castAndRenameOutput|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500]
[,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500],]
and for DSV2 in [in Analyzer.ResolveOutputRelation|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2239]
Spark puts effort in to reordering column names in line with what the output expects, but
merely casts any other types. This works nicely in a case where you try to e.g. write an int
into a double field, but goes wrong on complex datatypes if the types match up but the field
names do not.


> Corruption when writing data into a catalog table with a different struct schema
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-25943
>                 URL: https://issues.apache.org/jira/browse/SPARK-25943
>             Project: Spark
>          Issue Type: Bug
>          Components: Optimizer, SQL
>    Affects Versions: 2.3.2, 2.4.1, 2.5.0, 3.0.0
>            Reporter: James Baker
>            Priority: Major
>
> Suppose I have a catalog table with schema StructType(Seq(StructField("a", StructType(Seq(StructField("b",
DataTypes.StringType), StructField("c", DataTypes.StringType))).
> Suppose I now try to append a record to it:
> {code:java}
> {"a": {"c": "data1", "b": "data2"}}
> {code}
> That record will actually be appended as:
> {code:java}
> {"a": {"b": "data1", "c": "data2"}}
> {code}
> which is obviously not close to what the user wanted or expected (for me it silently
corrupted my data).
> It turns out that the user could provide a totally different record,
> {code:java}
> {"a": {"this column": "is totally different", "but": "the types match up"}}
> {code}
> and it'd still get written out, but as
> {code:java}
> {"a": {"b": "is totally different", "c": "the types match up"}}
> {code}
> This is because [in DDLPreprocessingUtils.castAndRenameOutput|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500]
[,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L500],]
Spark puts effort in to reordering column names in line with what the output expects, but
merely casts any other types. This works nicely in a case where you try to e.g. write an int
into a double field, but goes wrong on complex datatypes if the types match up but the field
names do not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message