spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srinath (JIRA)" <>
Subject [jira] [Commented] (SPARK-18209) More robust view canonicalization without full SQL expansion
Date Tue, 01 Nov 2016 22:12:58 GMT


Srinath commented on SPARK-18209:

A practical (positive) consequence of this is query expansion when we have nested views:
create table T(a int)
create view A as select * from T
view B = select * from A
As it stands, the definition of B is frozen at view creation time, so
drop view A
create view A as select * from T2
select * from B
would return data from T even though the definition of A has changed.
If we only expand view definition at query time, then the above would return data from T2

> More robust view canonicalization without full SQL expansion
> ------------------------------------------------------------
>                 Key: SPARK-18209
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Critical
> Spark SQL currently stores views by analyzing the provided SQL and then generating fully
expanded SQL out of the analyzed logical plan. This is actually a very error prone way of
doing it, because:
> 1. It is non-trivial to guarantee that the generated SQL is correct without being extremely
verbose, given the current set of operators.
> 2. We need extensive testing for all combination of operators.
> 3. Whenever we introduce a new logical plan operator, we need to be super careful because
it might break SQL generation. This is the main reason broadcast join hint has taken forever
to be merged because it is very difficult to guarantee correctness.
> Given the two primary reasons to do view canonicalization is to provide the context for
the database as well as star expansion, I think we can this through a simpler approach, by
taking the user given SQL, analyze it, and just wrap the original SQL with a SELECT clause
at the outer and store the database as a hint.
> For example, given the following view creation SQL:
> {code}
> CREATE TABLE my_table (id int, name string);
> CREATE VIEW my_view AS SELECT * FROM my_table WHERE id > 10;
> {code}
> We store the following SQL instead:
> {code}
> SELECT /*+ current_db: `my_db` */ id, name FROM (SELECT * FROM my_table WHERE id >
> {code}
> During parsing time, we expand the view along using the provided database context.
> (We don't need to follow exactly the same hint, as I'm merely illustrating the high level
approach here.)
> Note that there is a chance that the underlying base table(s)' schema change and the
stored schema of the view might differ from the actual SQL schema. In that case, I think we
should throw an exception at runtime to warn users. This exception can be controlled by a

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message