spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-18549) Failed to Uncache a View that References a Dropped Table.
Date Wed, 23 Nov 2016 17:16:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690765#comment-15690765
] 

Xiao Li edited comment on SPARK-18549 at 11/23/16 5:15 PM:
-----------------------------------------------------------

Are you suggesting name-based resolution? This is risky. You know, cache manager is cross
session. It is very hard for us to determine which temporary views, views, tables, or global
temporary tables are the original cached data.

Cross session unique ID might be needed. However, I am not sure this is a right direction.


BTW, this could become worse if we re-implement the views in Spark 2.2. 




was (Author: smilegator):
Are you suggesting name-based resolution? This is risky. You know, cache manager is cross
session. It is very hard for us to determine the original temporary views, views, tables,
or global temporary tables. 

Cross session unique ID might be needed. However, I am not sure this is a right direction.


BTW, this could become worse if we re-implement the views in Spark 2.2. 



> Failed to Uncache a View that References a Dropped Table.
> ---------------------------------------------------------
>
>                 Key: SPARK-18549
>                 URL: https://issues.apache.org/jira/browse/SPARK-18549
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.2
>            Reporter: Xiao Li
>            Priority: Critical
>
> {code}
>       spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
>       spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
>       sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
>       // Cache is empty at the beginning
>       assert(spark.sharedState.cacheManager.isEmpty)
>       sql("CACHE TABLE testView")
>       assert(spark.catalog.isCached("testView"))
>       // Cache is not empty
>       assert(!spark.sharedState.cacheManager.isEmpty)
> {code}
> {code}
>       // drop a table referenced by a cached view
>       sql("DROP TABLE jt1")
> -- So far everything is fine
>       // Failed to unache the view
>       val e = intercept[AnalysisException] {
>         sql("UNCACHE TABLE testView")
>       }.getMessage
>       assert(e.contains("Table or view not found: `default`.`jt1`"))
>       // We are unable to drop it from the cache
>       assert(!spark.sharedState.cacheManager.isEmpty)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message