hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2942) Aggregated Log Files should be compacted
Date Fri, 30 Jan 2015 19:51:35 GMT

     [ https://issues.apache.org/jira/browse/YARN-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Robert Kanter updated YARN-2942:
    Attachment: YARN-2942-preliminary.002.patch

I've just uploaded YARN-2942-preliminary.002.patch, which follows the v2 design doc.  It's
basically all there except for unit tests and ZK/Curator security stuff, which I'm still working

Here's some more technical info on the implementation:
- Compacted logs and are placed in the same directory as the aggregated logs, which has some
benefits, including
-- No need to duplicate directory structure
-- {{AggregatedLogDeletionService}} handles cleaning up old compacted logs without any changes
- {{CompactedAggregatedLogFormat}} has a Reader and Writer that handles the details of reading
and writing the compacted logs.
- To simplify some reading code in the {{AggregatedLogsBlock}}, I created a {{LogFormatReader}}
interface, which defines some common methods that both an {{AggregatedLogFormat.LogReader}}
and a {{CompactedAggregatedLogFormat.LogReader}} both have
- The {{AggregatedLogsBlock}} first tries to read from the compacted log file; if it can't
find it, or can't find the container in the index, or has some other problem, it will fallback
to the aggregated log and have the same behavior as before
-- The file formats for the aggregated logs and compacted logs are similar enough that the
{{AggregatedLogFormat.ContainerLogsReader}} can be used on either, so there's no new log file
parsing code for that
- Here's the process that the NM goes through (if compaction is enabled):
-- After the {{AppLogAggregatorImpl}} is done uploading aggregated log files, it will then
try to acquire the Curator lock for the current application
-- Then it will append its log file
-- Then it will delete its aggregated log file
-- Then it will release the lock

It would be great if I could get some feedback on the current patch so far.

> Aggregated Log Files should be compacted
> ----------------------------------------
>                 Key: YARN-2942
>                 URL: https://issues.apache.org/jira/browse/YARN-2942
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.6.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: CompactedAggregatedLogsProposal_v1.pdf, CompactedAggregatedLogsProposal_v2.pdf,
YARN-2942-preliminary.001.patch, YARN-2942-preliminary.002.patch
> Turning on log aggregation allows users to easily store container logs in HDFS and subsequently
view them in the YARN web UIs from a central place.  Currently, there is a separate log file
for each Node Manager.  This can be a problem for HDFS if you have a cluster with many nodes
as you’ll slowly start accumulating many (possibly small) files per YARN application.  The
current “solution” for this problem is to configure YARN (actually the JHS) to automatically
delete these files after some amount of time.  
> We should improve this by compacting the per-node aggregated log files into one log file
per application.

This message was sent by Atlassian JIRA

View raw message