hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-2942) Aggregated Log Files should be compacted
Date Tue, 09 Dec 2014 23:28:12 GMT
Robert Kanter created YARN-2942:

             Summary: Aggregated Log Files should be compacted
                 Key: YARN-2942
                 URL: https://issues.apache.org/jira/browse/YARN-2942
             Project: Hadoop YARN
          Issue Type: New Feature
    Affects Versions: 2.6.0
            Reporter: Robert Kanter
            Assignee: Robert Kanter

Turning on log aggregation allows users to easily store container logs in HDFS and subsequently
view them in the YARN web UIs from a central place.  Currently, there is a separate log file
for each Node Manager.  This can be a problem for HDFS if you have a cluster with many nodes
as you’ll slowly start accumulating many (possibly small) files per YARN application.  The
current “solution” for this problem is to configure YARN (actually the JHS) to automatically
delete these files after some amount of time.  

We should improve this by compacting the per-node aggregated log files into one log file per

This message was sent by Atlassian JIRA

View raw message