Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 786F817F93 for ; Sat, 2 May 2015 00:03:53 +0000 (UTC) Received: (qmail 67888 invoked by uid 500); 2 May 2015 00:03:53 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 67829 invoked by uid 500); 2 May 2015 00:03:53 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 67817 invoked by uid 99); 2 May 2015 00:03:53 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 May 2015 00:03:53 +0000 Date: Sat, 2 May 2015 00:03:53 +0000 (UTC) From: "Robert Kanter (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2942) Aggregated Log Files should be combined MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2942?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14524= 338#comment-14524338 ]=20 Robert Kanter commented on YARN-2942: ------------------------------------- I've been playing around with the LogAggregationStatus stuff and I think we= should be able to build on top of it. I'm working on a new design documen= t that I'll hopefully post sometime early next week. > Aggregated Log Files should be combined > --------------------------------------- > > Key: YARN-2942 > URL: https://issues.apache.org/jira/browse/YARN-2942 > Project: Hadoop YARN > Issue Type: New Feature > Affects Versions: 2.6.0 > Reporter: Robert Kanter > Assignee: Robert Kanter > Attachments: CombinedAggregatedLogsProposal_v3.pdf, CombinedAggre= gatedLogsProposal_v6.pdf, CompactedAggregatedLogsProposal_v1.pdf, Compacted= AggregatedLogsProposal_v2.pdf, ConcatableAggregatedLogsProposal_v4.pdf, Con= catableAggregatedLogsProposal_v5.pdf, YARN-2942-preliminary.001.patch, YARN= -2942-preliminary.002.patch, YARN-2942.001.patch, YARN-2942.002.patch, YARN= -2942.003.patch > > > Turning on log aggregation allows users to easily store container logs in= HDFS and subsequently view them in the YARN web UIs from a central place. = Currently, there is a separate log file for each Node Manager. This can b= e a problem for HDFS if you have a cluster with many nodes as you=E2=80=99l= l slowly start accumulating many (possibly small) files per YARN applicatio= n. The current =E2=80=9Csolution=E2=80=9D for this problem is to configure= YARN (actually the JHS) to automatically delete these files after some amo= unt of time. =20 > We should improve this by compacting the per-node aggregated log files in= to one log file per application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)