hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongwook Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3929) Uncleaning option for local app log files with log-aggregation feature
Date Sat, 18 Jul 2015 17:37:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632535#comment-14632535

Dongwook Kwon commented on YARN-3929:

Thanks Xuan for the information.
I quickly looked yarn.nodemanager.delete.debug-delay-sec and tested, it appears the setting
affect on DeletionService which means it will delay or not to delete all local files which
are supposed to be deleted by DeletionService?  I do want to keep application log for my own
backup/troubleshooting but not for other files for such as application's localization, usercache,
filecache, nmPrivate, spilled files etc, I would like to delete these as quick cycle as possible.
Please correct me if I was misunderstood about yarn.nodemanager.delete.debug-delay-sec
I couldn't find exact what I want, If there is any option that I can keep only application
log in local with log-aggregation feature, I would just use it and close this case.  

> Uncleaning option for local app log files with log-aggregation feature
> ----------------------------------------------------------------------
>                 Key: YARN-3929
>                 URL: https://issues.apache.org/jira/browse/YARN-3929
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: log-aggregation
>    Affects Versions: 2.4.0, 2.6.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>         Attachments: YARN-3929.01.patch
> Although it makes sense to delete local app log files once AppLogAggregator copied all
files into remote location(HDFS), I have some use cases that need to leave local app log files
after it's copied to HDFS. Mostly it's for own backup purpose. I would like to use log-aggregation
feature of YARN and want to back up app log files too. Without this option, files has to copy
from HDFS to local again. 

This message was sent by Atlassian JIRA

View raw message