hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13470) GenericTestUtils$LogCapturer is flaky
Date Tue, 16 Aug 2016 22:34:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423518#comment-15423518
] 

Chris Nauroth commented on HADOOP-13470:
----------------------------------------

Hello [~liuml07].  This patch unfortunately broke {{TestFileSystemOperationsWithThreads}}
in hadoop-azure.  It wasn't caught in pre-commit, because {{TestFileSystemOperationsWithThreads}}
doesn't execute unless the build environment is configured with Azure Storage credentials
(similar to hadoop-aws).

The failing tests have a few assertions that check for log messages containing specific thread
names:

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java#L285-L288

This is intended to validate that all expected threads in a parallelized operation performed
some work.  After {{LogCapturer}} switched to using a default {{PatternLayout}}, the thread
name is no longer included in the captured logs.

Possible solutions for this are either to revert the patch or change the instantiated {{PatternLayout}}
to use a pattern that includes thread name.  I scanned a few of our existing test log4j.properties
files, and there isn't a single consistent pattern used across all of them right now, so I
guess we'd just have to pick something reasonable and go with it.

Let me know your thoughts.  Thanks.

> GenericTestUtils$LogCapturer is flaky
> -------------------------------------
>
>                 Key: HADOOP-13470
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13470
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test, util
>    Affects Versions: 2.8.0
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>              Labels: reviewed
>             Fix For: 2.8.0
>
>         Attachments: HADOOP-13470.000.patch, HADOOP-13470.001.patch
>
>
> {{GenericTestUtils$LogCapturer}} is useful for assertions against service logs. However
it should be fixed in following aspects:
> # In the constructor, it uses the stdout appender's layout.
> {code}
> Layout layout = Logger.getRootLogger().getAppender("stdout").getLayout();
> {code}
> However, the stdout appender may be named "console" or alike which makes the constructor
throw NPE. Actually the layout does not matter and we can use a default pattern layout that
only captures application logs.
> # {{stopCapturing()}} method is not working. The major reason is that the {{appender}}
internal variable is never assigned and thus removing it to stop capturing makes no sense.
> # It does not support {{org.slf4j.Logger}} which is preferred to log4j in many modules.
> # There is no unit test for it.
> This jira is to address these.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message