spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system
Date Thu, 22 Dec 2016 20:05:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770967#comment-15770967
] 

Dongjoon Hyun edited comment on SPARK-18941 at 12/22/16 8:05 PM:
-----------------------------------------------------------------

Hi, first of all. your case is correct. I can reproduce your example. Thanks.

Currently, it seems to be intentional behavior because Spark assumes your table is EXTERNAL
when users give locations.

{code}
scala> sql("create table table_with_location(a int) stored as orc location '/tmp/table_with_location'")
scala> sql("desc extended table_with_location").show(false)
...
|# Detailed Table Information|CatalogTable(
	Table: `default`.`table_with_location`
	Owner: dhyun
	Created: Thu Dec 22 12:01:35 PST 2016
	Last Access: Wed Dec 31 16:00:00 PST 1969
	Type: EXTERNAL
	Schema: [StructField(a,IntegerType,true)]
	Provider: hive
	Properties: [transient_lastDdlTime=1482436895]
...
{code}

Let me try to make a PR for this.


was (Author: dongjoon):
Hi, first of all. your case is correct. I can reproduce your example. Thanks.

Currently, it seems to be intentional behavior because Spark assumes your table is EXTERNAL
when users give locations.

```
scala> sql("create table table_with_location(a int) stored as orc location '/tmp/table_with_location'")
res0: org.apache.spark.sql.DataFrame = []

scala> sql("desc extended table_with_location").show(false)
...
|# Detailed Table Information|CatalogTable(
	Table: `default`.`table_with_location`
	Owner: dhyun
	Created: Thu Dec 22 12:01:35 PST 2016
	Last Access: Wed Dec 31 16:00:00 PST 1969
	Type: EXTERNAL
	Schema: [StructField(a,IntegerType,true)]
	Provider: hive
	Properties: [transient_lastDdlTime=1482436895]
...
```

Let me try to make a PR for this.

> Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory
associated with the Hive table (not EXTERNAL table) from the HDFS file system
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-18941
>                 URL: https://issues.apache.org/jira/browse/SPARK-18941
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 2.0.2
>            Reporter: luat
>
> Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory
associated with the Hive table (not EXTERNAL table) from the HDFS file system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message