hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongwook Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3929) Uncleaning option for local app log files with log-aggregation feature
Date Sat, 18 Jul 2015 17:37:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632535#comment-14632535
] 

Dongwook Kwon commented on YARN-3929:
-------------------------------------

Thanks Xuan for the information.
I quickly looked yarn.nodemanager.delete.debug-delay-sec and tested, it appears the setting
affect on DeletionService which means it will delay or not to delete all local files which
are supposed to be deleted by DeletionService?  I do want to keep application log for my own
backup/troubleshooting but not for other files for such as application's localization, usercache,
filecache, nmPrivate, spilled files etc, I would like to delete these as quick cycle as possible.
Please correct me if I was misunderstood about yarn.nodemanager.delete.debug-delay-sec
I couldn't find exact what I want, If there is any option that I can keep only application
log in local with log-aggregation feature, I would just use it and close this case.  

> Uncleaning option for local app log files with log-aggregation feature
> ----------------------------------------------------------------------
>
>                 Key: YARN-3929
>                 URL: https://issues.apache.org/jira/browse/YARN-3929
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: log-aggregation
>    Affects Versions: 2.4.0, 2.6.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>         Attachments: YARN-3929.01.patch
>
>
> Although it makes sense to delete local app log files once AppLogAggregator copied all
files into remote location(HDFS), I have some use cases that need to leave local app log files
after it's copied to HDFS. Mostly it's for own backup purpose. I would like to use log-aggregation
feature of YARN and want to back up app log files too. Without this option, files has to copy
from HDFS to local again. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message