hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
Date Tue, 01 Aug 2017 01:07:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108233#comment-16108233
] 

Hadoop QA commented on HADOOP-14701:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 16s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 22s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 33s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 39s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 27s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 24s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 50s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 38s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 18s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 18s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 39s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 25s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 30s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 49s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m  1s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 29s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 10s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14701 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879743/HADOOP-14701.3.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 18ce03249b7d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ea56812 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12907/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12907/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
|
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12907/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Configuration can log misleading warnings about an attempt to override final parameter
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Andrew Sherman
>         Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch, HADOOP-14701.3.patch
>
>
> The class Configuration (in hadoop-common-project) is used to hold configuration properties.

> The method loadResources() is used to load properties from a list of default resources
into a parameter called  “properties” (which happens to always be the field called “properties”).

> This is done by calling the method loadResource() which is used to load properties from
a single resource into a parameter called  “properties” (which happens to always be the
field called “properties”). 
> This is done by calling loadProperty() which is used to load a single property into a
parameter called  “properties” (which is sometimes, but not always, the field called “properties”).

> In loadProperty()  there is some logic for dealing with properties that are marked as
final.
> (1) a final property is not set into the “properties” parameter.
> (2) a warning is printed for a final property if the value of the property with the same
name in the “properties” parameter is not equal to the new value, and the property name
is in the finalParameters field.
> When loadResource() is reading properties from a stream it creates a temporary Properties
object that it populates by calling loadProperty(). This is the case where  loadProperty()
has a  parameter called “properties” which is not the same as the field called “properties”).

> When loadProperty() is called on a final property then because of (1) it is not set into
the “properties” parameter. But sometimes the property name is already in the finalParameters
field and now (2) the warning is triggered because the property is not present in the “properties”
parameter (although it IS present in the “properties” field). The warning is misleading
and confusing. 
> h3. Possible Fix
> One way of looking at the problem is that loadProperty is using the finalParameters field
in combination with the “properties” parameter. Change the code to only generate a warning
for a final property if the value of the property with the same name in the “properties”
field is not equal to the new value, and the property name is in the finalParameters field.
> I’ve made a patch containing this tiny change.
> Note that I have done very minimal testing of this fix
> h3. Example warning:
> 2017-07-25 12:14:24,639 WARN  org.apache.hadoop.conf.Configuration: [HiveServer2-Background-Pool:
Thread-77]: hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an
attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message