hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15015) TestConfigurationFieldsBase to use SLF4J for logging
Date Fri, 03 Nov 2017 16:41:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-15015:
------------------------------------
    Attachment: HADOOP-15015-001.patch

Patch 001

1. main log for normal output. special xml and config ones for debugging at that level.
1. moved all the logging
1. Deleted the old boolean keys...nothing downstream was using them.
1. Added empty sets to the relevant fields, so we can eliminate all the != null checks
1. Sort things for output by feeding sets through TreeSet before iterating through the values.
1. Assertion errors now include error text.
 1. Went with the IDE's java 8 migrations to leaner code all round

Example
{code}
Failed tests: 
  TestMapreduceConfigFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass:542
mapred-default.xml has 2 properties missing in  interface org.apache.hadoop.mapreduce.MRJobConfig
 interface org.apache.hadoop.mapreduce.MRConfig  class org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig
 class org.apache.hadoop.mapred.ShuffleHandler  class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
 class org.apache.hadoop.mapreduce.lib.input.FileInputFormat  class org.apache.hadoop.mapreduce.Job
 class org.apache.hadoop.mapreduce.lib.input.NLineInputFormat  class org.apache.hadoop.mapred.JobConf
 class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter Entries: mapreduce.outputcommitter.factory.class
 mapreduce.outputcommitter.factory.scheme.s3a expected:<0> but was:<2>
{code}

Getting that list of missing entries is key, as now Jenkins will include its list of problems
in all its failed tests. You shouldn't need to look at the output logs to debug basic problems.



> TestConfigurationFieldsBase to use SLF4J for logging
> ----------------------------------------------------
>
>                 Key: HADOOP-15015
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15015
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf, test
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Bharat Viswanadham
>         Attachments: HADOOP-15015-001.patch
>
>
> {{TestConfigurationFieldsBase}} has a protected "configDebug" field used to turn logging
on/off
> {code}
>       if (configDebug) {
>         System.out.println("Field: " + f);
>       }
> {code}
> Presumably its there to allow people with code access to debug their classes. But if
we switch to SLF4J you get controllable logging @ runtime.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message