Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1638417805.1211564155731.JavaMail.jira@brutus>
Date: Fri, 23 May 2008 10:35:55 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3439) TaskTracker.addDiagnostics(String
 file, int num, String tag) could exit early if num==0
In-Reply-To: <1958799659.1211550475773.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599442#action_12599442 ] 

Doug Cutting commented on HADOOP-3439:
--------------------------------------

> loads in a conf option (that is not in hadoop-default, incidentally) 

The rule for whether things belong in hadoop-default.xml or not is whether or not they are intended to be overridden in hadoop-site.xml.  Many parameters are only intended to be set by code, adding these to hadoop-default.xml just clutters what's primarily meant to be documentation.  Parameters meant to be set only by code should have static accessor methods on a relevant class, e.g., Foo#setFoo(Configuration c, String value).  Also, it's reasonable to leave out of hadoop-default.xml debugging parameters that are intended only for use by developers, not by end users.

That's been the (unwritten?) policy.  Does it make sense?  If so, perhaps we should record it somewhere...

> TaskTracker.addDiagnostics(String file, int num, String tag) could exit early if num==0
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3439
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> When a TaskTracker job finishes,  taskFinished() is invoked. 
> as part of its work it
>  1. loads in a conf option (that is not in hadoop-default, incidentally) , mapred.debug.out.lines , default value -1;
>  2. calls addDiagnostics passing in that line count
> addDiagnostics either builds a string buffer of all the output, or creates a linear array of lines and runs adds them, shuffling them up if there are more lines than expected. 
> This is all unneeded if the number of lines to print == 0; the entire reading in of the output file can be skipped. This may speed up termination slightly on a run with a large output file and mapred.debug.out.lines ==0. 
> Note also that a circular buffer would handle the lines>0 problem without having to copy all the strings around.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.