Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 4 Nov 2016 17:41:58 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13006051.1474322132000.192649.1478281318572@Atlassian.JIRA>
In-Reply-To: <JIRA.13006051.1474322132000@Atlassian.JIRA>
References: <JIRA.13006051.1474322132000@Atlassian.JIRA> <JIRA.13006051.1474322132893@arcas>
Subject: [jira] [Commented] (YARN-5658) YARN should have a hook to delete a
 path from HDFS when an application ends
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 04 Nov 2016 17:42:00 -0000


    [ https://issues.apache.org/jira/browse/YARN-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637118#comment-15637118 ] 

Vinod Kumar Vavilapalli commented on YARN-5658:
-----------------------------------------------

I agree with [~rohithsharma]. This is a dup of YARN-2261.

bq. Spark could adapt their AM to use YARN-2261, but Hive couldn't
Assuming Hive is running on YARN through one of MR, Tez, Spark, one of those AMs should do this cleanup. If not, the cleanup should happen in HiveServer which is the long running client for Hive.

We don't want to run any code on the ResourceManager.

If we run it on a NodeManager, that's the same as a cleanup-container (YARN-2261).

> YARN should have a hook to delete a path from HDFS when an application ends
> ---------------------------------------------------------------------------
>
>                 Key: YARN-5658
>                 URL: https://issues.apache.org/jira/browse/YARN-5658
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>
> There are many cases when a client uploads data to HDFS and then needs to subsequently clean it up, such as with the distributed cache.  It would be helpful if YARN would do that cleanup automatically on job completion.
> The hook could be generic to an URI supported by {{FileSystem}}.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org