Return-Path: 
 <yarn-issues-return-16155-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E9E2C109B3
	for <apmail-hadoop-yarn-issues-archive@minotaur.apache.org>;
 Mon,  7 Oct 2013 17:55:53 +0000 (UTC)
Received: (qmail 21176 invoked by uid 500); 7 Oct 2013 17:55:48 -0000
Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org
Received: (qmail 20988 invoked by uid 500); 7 Oct 2013 17:55:45 -0000
Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:yarn-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:yarn-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:yarn-issues@hadoop.apache.org>
List-Id: <yarn-issues.hadoop.apache.org>
Reply-To: yarn-issues@hadoop.apache.org
Delivered-To: mailing list yarn-issues@hadoop.apache.org
Received: (qmail 20917 invoked by uid 99); 7 Oct 2013 17:55:43 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Oct 2013 17:55:43 +0000
Date: Mon, 7 Oct 2013 17:55:43 +0000 (UTC)
From: "Paul Han (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12666407.1377900967841.26666.1381168543528@arcas>
In-Reply-To: <JIRA.12666407.1377900967841@arcas>
References: <JIRA.12666407.1377900967841@arcas>
Subject: [jira] [Commented] (YARN-1130) Improve the log flushing for tasks
 when mapred.userlog.limit.kb is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788354#comment-13788354 ] 

Paul Han commented on YARN-1130:
--------------------------------

Thanks Arun for picking this up! 

I've made some changes to simplify the fix to do flush of logging with "wait". That should get rid of those outofmemory errors due to earlier addition of the asynchronous "flush" thread.  

I'll look into the other unit test failures. At first glance,  they don't seem to directly relate to the changes submitted. :(

> Improve the log flushing for tasks when mapred.userlog.limit.kb is set
> ----------------------------------------------------------------------
>
>                 Key: YARN-1130
>                 URL: https://issues.apache.org/jira/browse/YARN-1130
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.0.5-alpha
>            Reporter: Paul Han
>            Assignee: Paul Han
>             Fix For: 2.0.5-alpha
>
>         Attachments: YARN-1130.patch, YARN-1130.patch, YARN-1130.patch
>
>
> When userlog limit is set with something like this:
> {code}
> <property>
> <name>mapred.userlog.limit.kb</name>
> <value>2048</value>
> <description>The maximum size of user-logs of each task in KB. 0 disables the cap.
> </description>
> </property>
> {code}
> the log entry will be truncated randomly for the jobs.
> The log size is left between 1.2MB to 1.6MB.
> Since the log is already limited, avoid the log truncation is crucial for user.
> The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it.
> Will propose a patch to add a flush mechanism and also flush the log when task is done.  



--
This message was sent by Atlassian JIRA
(v6.1#6144)