From yarn-issues-return-47453-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 11 02:00:14 2015 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C980110495 for ; Wed, 11 Feb 2015 02:00:14 +0000 (UTC) Received: (qmail 34687 invoked by uid 500); 11 Feb 2015 02:00:14 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 34649 invoked by uid 500); 11 Feb 2015 02:00:14 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 34634 invoked by uid 99); 11 Feb 2015 02:00:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Feb 2015 02:00:14 +0000 Date: Wed, 11 Feb 2015 02:00:14 +0000 (UTC) From: "Hadoop QA (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2942) Aggregated Log Files should be compacted MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2942?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14315= 405#comment-14315405 ]=20 Hadoop QA commented on YARN-2942: --------------------------------- {color:red}-1 overall{color}. Here are the results of testing the latest a= ttachment=20 http://issues.apache.org/jira/secure/attachment/12697901/YARN-2942.002.pa= tch against trunk revision d5855c0. {color:green}+1 @author{color}. The patch does not contain any @author= tags. {color:green}+1 tests included{color}. The patch appears to include 3 = new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the = total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning mess= ages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:e= clipse. {color:red}-1 findbugs{color}. The patch appears to cause Findbugs (ve= rsion 2.0.3) to fail. {color:green}+1 release audit{color}. The applied patch does not incre= ase the total number of release audit warnings. {color:red}-1 core tests{color}. The test build failed in hadoop-yarn-= project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoo= p-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-ya= rn/hadoop-yarn-api=20 Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6586//test= Report/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6586//co= nsole This message is automatically generated. > Aggregated Log Files should be compacted > ---------------------------------------- > > Key: YARN-2942 > URL: https://issues.apache.org/jira/browse/YARN-2942 > Project: Hadoop YARN > Issue Type: New Feature > Affects Versions: 2.6.0 > Reporter: Robert Kanter > Assignee: Robert Kanter > Attachments: CompactedAggregatedLogsProposal_v1.pdf, CompactedAgg= regatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch, YARN-2942-prel= iminary.002.patch, YARN-2942.001.patch, YARN-2942.002.patch, YARN-2942.003.= patch > > > Turning on log aggregation allows users to easily store container logs in= HDFS and subsequently view them in the YARN web UIs from a central place. = Currently, there is a separate log file for each Node Manager. This can b= e a problem for HDFS if you have a cluster with many nodes as you=E2=80=99l= l slowly start accumulating many (possibly small) files per YARN applicatio= n. The current =E2=80=9Csolution=E2=80=9D for this problem is to configure= YARN (actually the JHS) to automatically delete these files after some amo= unt of time. =20 > We should improve this by compacting the per-node aggregated log files in= to one log file per application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)