hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5672) Provide optional RollingFileAppender for container log4j (syslog)
Date Mon, 13 Jan 2014 07:41:57 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869335#comment-13869335
] 

Hadoop QA commented on MAPREDUCE-5672:
--------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12622583/MAPREDUCE-5672.v07.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

                  org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4316//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4316//console

This message is automatically generated.

> Provide optional RollingFileAppender for container log4j (syslog)
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-5672
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>    Affects Versions: 2.2.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5672.v01.patch, MAPREDUCE-5672.v02.patch, MAPREDUCE-5672.v03.patch,
MAPREDUCE-5672.v04.patch, MAPREDUCE-5672.v05.patch, MAPREDUCE-5672.v06.patch, MAPREDUCE-5672.v07.patch,
MAPREDUCE-5672.v07.patch, Screen Shot 2013-12-05 at 3.21.02 PM.png, Screen Shot 2013-12-05
at 3.23.33 PM.png
>
>
> This JIRA is an alternative take on YARN-1130
> We propose providing an option of using a RollingFileAppender(RFA)-based implementation
of container log appender as means of log size control via mapreduce.task.userlog.limit.kb.

> The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA. In addition
yarn.app.mapreduce.container.log.backups (task attempt containers) and yarn.app.mapreduce.am.log.backups
(MR-AM) are passed as maxBackupIndex.
> Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender (CRLA) co-exist.
CLA is the default. CRLA is chosen when  mapreduce.task.userlog.limit.kb > 0 &&
*.backups > 0.
> Pros: 
> 1) CRLA output is visible in UI right away. CLA output with mapreduce.task.userlog.limit.kb
> 0 is not visible until the task attempt finishes that prevents timely diagnostics. 
> 2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb, no space
is taken from the JVM heap.
> 3) No UI impact, since YARN is already designed to deal with any log name beyond stderr/out,
syslog, debug.out, profile.out
> Cons:
> 1) if the logging is excessive there will be more local filesystem metadata I/O due to
roll. That should be negligible in the grand scheme.
> Furthermore, to improve log consistency and completeness in the case of JVM crashes and
SIGTERMing by NM, we propose to restore the MRv1 behavior of periodic log syncing (every 5s)
and having log sync as part of a shutdown hook.
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message