spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24170) [Spark SQL] json file format is not dropped after dropping table
Date Mon, 07 May 2018 15:59:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466071#comment-16466071
] 

Xiao Li commented on SPARK-24170:
---------------------------------

They are external tables when you specify the path in CREATE TABLE. Thus, the files will not
be dropped. 

> [Spark SQL] json file format is not dropped after dropping table
> ----------------------------------------------------------------
>
>                 Key: SPARK-24170
>                 URL: https://issues.apache.org/jira/browse/SPARK-24170
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>         Environment: OS: SUSE 11
> Spark Version: 2.3
>            Reporter: ABHISHEK KUMAR GUPTA
>            Priority: Major
>
> Steps:
>  # Launch spark-sql --master yarn
>  #  create table json(name STRING, age int, gender string, id INT) using org.apache.spark.sql.json
options(path "hdfs:///user/testdemo/");
>  # Execute the below SQL queries 
> INSERT into json
> SELECT 'Shaan',21,'Male',1
> UNION ALL
> SELECT 'Xing',20,'Female',11
> UNION ALL
> SELECT 'Mile',4,'Female',20
> UNION ALL
> SELECT 'Malan',10,'Male',9;
> Below 4 json file format created 
> BLR1000023111:/opt/Antsecure/install/hadoop/namenode/bin # ./hdfs dfs -ls /user/testdemo
> Found 14 items
> -rw-r--r-- 3 spark hadoop 0 2018-04-26 17:44 /user/testdemo/_SUCCESS
> -rw-r--r-- 3 spark hadoop 4802 2018-04-24 18:20 /user/testdemo/customer1.csv
> -rw-r--r-- 3 spark hadoop 92 2018-04-26 17:02 /user/testdemo/json1.txt
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32 /user/testdemo/part-00000-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44 /user/testdemo/part-00000-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 51 2018-04-26 17:32 /user/testdemo/part-00001-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 51 2018-04-26 17:44 /user/testdemo/part-00001-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 50 2018-04-26 17:32 /user/testdemo/part-00002-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 50 2018-04-26 17:44 /user/testdemo/part-00002-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32 /user/testdemo/part-00003-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
> -rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44 /user/testdemo/part-00003-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
>  
> Issue is:
> Now executed below drop command
> spark-sql> drop table json;
>  
> Table dropped successfully but json file still present in the path  /user/testdemo



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message