hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongwook Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3929) Uncleaning option for local app log files with log-aggregation feature
Date Mon, 20 Jul 2015 22:45:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634202#comment-14634202
] 

Dongwook Kwon commented on YARN-3929:
-------------------------------------

The reason is we already had similar tool as log-aggregator out of hadoop, not only for YARN,
it was designed for Hadoop 1 which didn't have native log-aggregation feature, in our cluster,
 each node has daemon that periodically checks application log in local and push to S3, it
works fine even with 2000 nodes, the issue we have now is with YARN's log-aggregation, as
you can imagine, 2 systems tries to do the same things, and other internal users want to use
YARN's log-aggregation for such as HUE or "yarn logs --applicationId" command, and we still
need to support Hadoop 1, so whenever  cluster turns on YARN's log-aggregation, we don't have
application log for troubleshooting. This has been an issue for long and simple solution for
our team is making this optional as I suggested, I agree, for most of use cases, it may not
be useful, so I make default as cleaning up and make sure test catch it.

> Uncleaning option for local app log files with log-aggregation feature
> ----------------------------------------------------------------------
>
>                 Key: YARN-3929
>                 URL: https://issues.apache.org/jira/browse/YARN-3929
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: log-aggregation
>    Affects Versions: 2.4.0, 2.6.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>         Attachments: YARN-3929.02.patch
>
>
> Although it makes sense to delete local app log files once AppLogAggregator copied all
files into remote location(HDFS), I have some use cases that need to leave local app log files
after it's copied to HDFS. Mostly it's for own backup purpose. I would like to use log-aggregation
feature of YARN and want to back up app log files too. Without this option, files has to copy
from HDFS to local again. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message