spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Otto (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works
Date Fri, 06 Apr 2018 16:50:00 GMT
Andrew Otto created SPARK-23890:
-----------------------------------

             Summary: Hive ALTER TABLE CHANGE COLUMN for struct type no longer works
                 Key: SPARK-23890
                 URL: https://issues.apache.org/jira/browse/SPARK-23890
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Andrew Otto


As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE COLUMN commands
to Hive.  This was expanded in [https://github.com/apache/spark/pull/12714] to allow for
those commands if they only change the column comment.

Wikimedia has been evolving Parquet backed Hive tables with data originally come from JSON
data by adding newly found columns to the Hive table schema, via a Spark job we call 'Refine'. 
We do this by recursively merging an input DataFrame schema with a Hive table DataFrame schema,
finding new fields, and then issuing an ALTER TABLE statement to add the columns.  However,
because we allow for nested data types in the incoming JSON data, we make extensive use of
struct type fields.  In order to add newly detected fields in a nested data type, we must
alter the struct column and append the nested struct field.  This requires CHANGE COLUMN
that alters the column type.  In reality, the 'type' of the column is not changing, it just
just a new field being added to the struct, but to SQL, this looks like a type change.

We were about to upgrade to Spark 2 but this new restriction in SQL DDL that can be sent to
Hive will block us.  I believe this is fixable by adding an exception in [command/ddl.scala|https://github.com/apache/spark/blob/v2.3.0/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L294-L325] to
allow ALTER TABLE CHANGE COLUMN with a new type, if the original type and destination type
are both struct types, and the destination type only adds new fields.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message