hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
Date Sat, 14 Sep 2013 01:11:52 GMT

    [ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13767279#comment-13767279
] 

Hadoop QA commented on YARN-1130:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12603162/YARN-1130.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

                  org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
                  org.apache.hadoop.mapred.TestNetworkedJob
                  org.apache.hadoop.mapred.TestClusterMRNotification
                  org.apache.hadoop.mapred.TestJobCounters
                  org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
                  org.apache.hadoop.mapred.TestMiniMRClasspath
                  org.apache.hadoop.mapred.TestBlockLimits
                  org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
                  org.apache.hadoop.mapred.TestMiniMRChildTask
                  org.apache.hadoop.mapreduce.security.TestMRCredentials
                  org.apache.hadoop.mapreduce.v2.TestNonExistentJob
                  org.apache.hadoop.mapreduce.v2.TestRMNMInfo
                  org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
                  org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
                  org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
                  org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
                  org.apache.hadoop.mapred.TestJobCleanup
                  org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
                  org.apache.hadoop.mapred.TestReduceFetch
                  org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
                  org.apache.hadoop.mapred.TestMerge
                  org.apache.hadoop.mapreduce.v2.TestMRJobs
                  org.apache.hadoop.mapreduce.TestMRJobClient
                  org.apache.hadoop.mapred.TestTaskCommit
                  org.apache.hadoop.mapreduce.TestChild
                  org.apache.hadoop.mapred.TestJobName
                  org.apache.hadoop.mapred.TestLazyOutput
                  org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
                  org.apache.hadoop.mapreduce.v2.TestUberAM
                  org.apache.hadoop.mapred.TestMiniMRClientCluster
                  org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
                  org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
                  org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
                  org.apache.hadoop.ipc.TestSocketFactory
                  org.apache.hadoop.mapred.TestJobSysDirWithDFS

                                      The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.conf.TestNoDefaultsJobConf

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1929//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1929//console

This message is automatically generated.
                
> Improve the log flushing for tasks when mapred.userlog.limit.kb is set
> ----------------------------------------------------------------------
>
>                 Key: YARN-1130
>                 URL: https://issues.apache.org/jira/browse/YARN-1130
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.0.5-alpha
>            Reporter: Paul Han
>             Fix For: 2.0.5-alpha
>
>         Attachments: YARN-1130.patch
>
>
> When userlog limit is set with something like this:
> {code}
> <property>
> <name>mapred.userlog.limit.kb</name>
> <value>2048</value>
> <description>The maximum size of user-logs of each task in KB. 0 disables the cap.
> </description>
> </property>
> {code}
> the log entry will be truncated randomly for the jobs.
> The log size is left between 1.2MB to 1.6MB.
> Since the log is already limited, avoid the log truncation is crucial for user.
> The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is
that log entries will not flush to file until the container shutdown and logmanager close
all appenders. If user likes to see the log during task execution, it doesn't support it.
> Will propose a patch to add a flush mechanism and also flush the log when task is done.
 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message