spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lufei (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21822) When insert Hive Table is finished, it is better to clean out the tmpLocation dir
Date Thu, 24 Aug 2017 06:55:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

lufei updated SPARK-21822:
--------------------------
    Description: 
When insert Hive Table is finished, it is better to clean out the tmpLocation dir(the temp
directories like ".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
or "/tmp/hive/..." for an old spark version).
Otherwise, when lots of spark job are executed, millions of temporary directories are left
in HDFS. And these temporary directories can only be deleted by the maintainer through the
shell script.

  was:
When insert Hive Table is finished, it is better to clean out the tmpLocation dir(the temp
directorys like ".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
or "/tmp/hive/..." for an old spark version).
Otherwise, when lots of spark job are executed, millions of temporary directories are left
in HDFS. And these temporary directories can only be deleted by the maintainer through the
shell script.


> When insert Hive Table is finished, it is better to clean out the tmpLocation dir
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-21822
>                 URL: https://issues.apache.org/jira/browse/SPARK-21822
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: lufei
>
> When insert Hive Table is finished, it is better to clean out the tmpLocation dir(the
temp directories like ".hive-staging_hive_2017-08-19_10-56-01_540_5448395226195533570-9/-ext-10000"
or "/tmp/hive/..." for an old spark version).
> Otherwise, when lots of spark job are executed, millions of temporary directories are
left in HDFS. And these temporary directories can only be deleted by the maintainer through
the shell script.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message