Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 1 Aug 2017 01:07:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13090914.1501281425000.57674.1501549620811@Atlassian.JIRA>
In-Reply-To: <JIRA.13090914.1501281425000@Atlassian.JIRA>
References: <JIRA.13090914.1501281425000@Atlassian.JIRA> <JIRA.13090914.1501281425730@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14701) Configuration can log misleading
 warnings about an attempt to override final parameter
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 01 Aug 2017 01:07:09 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14701?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16=
108233#comment-16108233 ]=20

Hadoop QA commented on HADOOP-14701:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 16s=
{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0=
m  0s{color} | {color:green} The patch does not contain any @author tags. {=
color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}=
  0m  0s{color} | {color:green} The patch appears to include 1 new or modif=
ied test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}=
 14m 22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13=
m 33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}=
  0m 39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1=
m 27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  =
1m 24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0=
m 50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}=
  0m 38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10=
m 18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m =
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}=
  0m 39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1=
m 25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}=
  0m  0s{color} | {color:green} The patch has no whitespace issues. {color}=
 |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  =
1m 30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0=
m 49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m  1s{col=
or} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}=
  0m 29s{color} | {color:green} The patch does not generate ASF License war=
nings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 10s{colo=
r} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14701 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/1287974=
3/HADOOP-14701.3.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsit=
e  unit  findbugs  checkstyle  |
| uname | Linux 18ce03249b7d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 1=
1:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provide=
d.sh |
| git revision | trunk / ea56812 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/12907/artifac=
t/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/1290=
7/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project=
/hadoop-common |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/129=
07/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.


> Configuration can log misleading warnings about an attempt to override fi=
nal parameter
> -------------------------------------------------------------------------=
-------------
>
>                 Key: HADOOP-14701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Andrew Sherman
>         Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch, HADOOP-1=
4701.3.patch
>
>
> The class Configuration (in hadoop-common-project) is used to hold config=
uration properties.=20
> The method loadResources() is used to load properties from a list of defa=
ult resources into a parameter called  =E2=80=9Cproperties=E2=80=9D (which =
happens to always be the field called =E2=80=9Cproperties=E2=80=9D).=20
> This is done by calling the method loadResource() which is used to load p=
roperties from a single resource into a parameter called  =E2=80=9Cproperti=
es=E2=80=9D (which happens to always be the field called =E2=80=9Cpropertie=
s=E2=80=9D).=20
> This is done by calling loadProperty() which is used to load a single pro=
perty into a parameter called  =E2=80=9Cproperties=E2=80=9D (which is somet=
imes, but not always, the field called =E2=80=9Cproperties=E2=80=9D).=20
> In loadProperty()  there is some logic for dealing with properties that a=
re marked as final.
> (1) a final property is not set into the =E2=80=9Cproperties=E2=80=9D par=
ameter.
> (2) a warning is printed for a final property if the value of the propert=
y with the same name in the =E2=80=9Cproperties=E2=80=9D parameter is not e=
qual to the new value, and the property name is in the finalParameters fiel=
d.
> When loadResource() is reading properties from a stream it creates a temp=
orary Properties object that it populates by calling loadProperty(). This i=
s the case where  loadProperty() has a  parameter called =E2=80=9Cpropertie=
s=E2=80=9D which is not the same as the field called =E2=80=9Cproperties=E2=
=80=9D).=20
> When loadProperty() is called on a final property then because of (1) it =
is not set into the =E2=80=9Cproperties=E2=80=9D parameter. But sometimes t=
he property name is already in the finalParameters field and now (2) the wa=
rning is triggered because the property is not present in the =E2=80=9Cprop=
erties=E2=80=9D parameter (although it IS present in the =E2=80=9Cpropertie=
s=E2=80=9D field). The warning is misleading and confusing.=20
> h3. Possible Fix
> One way of looking at the problem is that loadProperty is using the final=
Parameters field in combination with the =E2=80=9Cproperties=E2=80=9D param=
eter. Change the code to only generate a warning for a final property if th=
e value of the property with the same name in the =E2=80=9Cproperties=E2=80=
=9D field is not equal to the new value, and the property name is in the fi=
nalParameters field.
> I=E2=80=99ve made a patch containing this tiny change.
> Note that I have done very minimal testing of this fix
> h3. Example warning:
> 2017-07-25 12:14:24,639 WARN  org.apache.hadoop.conf.Configuration: [Hive=
Server2-Background-Pool: Thread-77]: hdfs://asherman-cdh6-subset-parcel-1.g=
ce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an a=
ttempt to override final parameter: mapreduce.job.end-notification.max.retr=
y.interval;  Ignoring.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org