spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Serge Smertin (JIRA)" <>
Subject [jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table
Date Thu, 21 Sep 2017 12:31:00 GMT


Serge Smertin commented on SPARK-18727:

in one of the use-cases for project in [#comment-15987668] by [~simeons] - adding fields to
nested _struct_ fields. application is built the way that parquet files are created/partitioned
outside of Spark and only new columns might be added. Again, mostly within couple of nested

I don't know all potential implications of the idea, but can we just use the last element
of selected files instead of the first one, as long as the FileStatus [list is already sorted
by path lexicographically|]?
It easier to guarantee that only new columns would be added over the time. And the following
code change doesn't seem to be huge deviation from current behavior, thus tremendously saving
time compared to {{spark.sql.parquet.mergeSchema=true}}:

{code:java} // ParquetFileFormat.scala (lines 232..240)

/cc [] [~xwu0226] 

> Support schema evolution as new files are inserted into table
> -------------------------------------------------------------
>                 Key: SPARK-18727
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Eric Liang
>            Priority: Critical
> Now that we have pushed partition management of all tables to the catalog, one issue
for scalable partition handling remains: handling schema updates.
> Currently, a schema update requires dropping and recreating the entire table, which does
not scale well with the size of the table.
> We should support updating the schema of the table, either via ALTER TABLE, or automatically
as new files with compatible schemas are appended into the table.
> cc [~rxin]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message