hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5658) YARN should have a hook to delete a path from HDFS when an application ends
Date Fri, 04 Nov 2016 17:41:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637118#comment-15637118

Vinod Kumar Vavilapalli commented on YARN-5658:

I agree with [~rohithsharma]. This is a dup of YARN-2261.

bq. Spark could adapt their AM to use YARN-2261, but Hive couldn't
Assuming Hive is running on YARN through one of MR, Tez, Spark, one of those AMs should do
this cleanup. If not, the cleanup should happen in HiveServer which is the long running client
for Hive.

We don't want to run any code on the ResourceManager.

If we run it on a NodeManager, that's the same as a cleanup-container (YARN-2261).

> YARN should have a hook to delete a path from HDFS when an application ends
> ---------------------------------------------------------------------------
>                 Key: YARN-5658
>                 URL: https://issues.apache.org/jira/browse/YARN-5658
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
> There are many cases when a client uploads data to HDFS and then needs to subsequently
clean it up, such as with the distributed cache.  It would be helpful if YARN would do that
cleanup automatically on job completion.
> The hook could be generic to an URI supported by {{FileSystem}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message