hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9017) Clean up temp files of RSC [Spark Branch]
Date Sat, 13 Dec 2014 00:11:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245020#comment-14245020
] 

Xuefu Zhang commented on HIVE-9017:
-----------------------------------

To clarify, when Spark lunched multiple executors in one host for one application, these executors
share the same JVM, right? At least that's my understanding. On the same host, there may be
other JVMs, but they will be for different applications. Different JVMs, and thus different
applications, shouldn't share the cache libs or data. That's my understanding, but I could
be bogus on this.

I can understand that Spark doesn't want each executor of an application to download the same
files. All executors in one JVM can share one copy of the files, as these executors are for
one application only. That's what I think SPARK-2713 is for.

> Clean up temp files of RSC [Spark Branch]
> -----------------------------------------
>
>                 Key: HIVE-9017
>                 URL: https://issues.apache.org/jira/browse/HIVE-9017
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>
> Currently RSC will leave a lot of temp files in {{/tmp}}, including {{*_lock}}, {{*_cache}},
{{spark-submit.*.properties}}, etc.
> We should clean up these files or it will exhaust disk space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message