hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6871) When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
Date Fri, 25 May 2012 13:56:23 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283422#comment-13283422
] 

Harsh J commented on HADOOP-6871:
---------------------------------

{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.ha.TestHealthMonitor
{quote}

This was perhaps an issue on trunk back then. Passes for me locally:

{code}
➜  hadoop-common  svn diff                                  
Index: src/test/java/org/apache/hadoop/conf/TestConfiguration.java
===================================================================
--- src/test/java/org/apache/hadoop/conf/TestConfiguration.java	(revision 1342610)
+++ src/test/java/org/apache/hadoop/conf/TestConfiguration.java	(working copy)
@@ -999,6 +999,15 @@
         "Not returning expected number of classes. Number of returned classes ="
             + classes.length, 0, classes.length);
   }
+
+  public void testInvalidSubstitutation() {
+    String key = "test.random.key";
+    String keyExpression = "${" + key + "}";
+    Configuration configuration = new Configuration();
+    configuration.set(key, keyExpression);
+    String value = configuration.get(key);
+    assertTrue("Unexpected value " + value, value.equals(keyExpression));
+  }
   
   public static void main(String[] argv) throws Exception {
     junit.textui.TestRunner.main(new String[]{
Index: src/main/java/org/apache/hadoop/conf/Configuration.java
===================================================================
--- src/main/java/org/apache/hadoop/conf/Configuration.java	(revision 1342610)
+++ src/main/java/org/apache/hadoop/conf/Configuration.java	(working copy)
@@ -617,7 +617,13 @@
     }
     Matcher match = varPat.matcher("");
     String eval = expr;
+    Set<String> evalSet = new HashSet<String>();
     for(int s=0; s<MAX_SUBST; s++) {
+      if (evalSet.contains(eval)) {
+        // Cyclic resolution pattern detected. Return current expression.
+        return eval;
+      }
+      evalSet.add(eval);
       match.reset(eval);
       if (!match.find()) {
         return eval;
➜  hadoop-common  mvn clean install -Dtest=TestConfiguration,TestHealthMonitor
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-common ---
[INFO] Deleting /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test-dir
    [mkdir] Created dir: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test/data
[INFO] Executed tasks
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ hadoop-common ---
[INFO] Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-sources/java
added.
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-test-source (add-test-source) @ hadoop-common
---
[INFO] Test Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java
added.
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (save-version) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-common ---
[INFO] Wrote classpath file '/Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes/mrapp-generated-classpath'.
[INFO] 
[INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-common ---
[INFO] Using default encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hadoop-common ---
[INFO] Compiling 629 source files to /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes
[INFO] 
[INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-test-sources) @ hadoop-common ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (compile-test-proto) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (generate-test-sources) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-common
---
[INFO] Using default encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-common ---
[INFO] Executing tasks

main:
   [delete] Deleting directory /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test/data
    [mkdir] Created dir: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test/data
    [mkdir] Created dir: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/log
     [copy] Copying 5 files to /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test-classes
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hadoop-common ---
[INFO] Compiling 309 source files to /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ hadoop-common ---
[INFO] Surefire report directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.conf.TestConfiguration
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.672 sec
Running org.apache.hadoop.ha.TestHealthMonitor
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.291 sec

Results :

Tests run: 40, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (prepare-jar) @ hadoop-common ---
[INFO] Building jar: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:test-jar (prepare-test-jar) @ hadoop-common ---
[INFO] Building jar: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] >>> maven-source-plugin:2.1.2:jar (default) @ hadoop-common >>>
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ hadoop-common ---
[INFO] Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-sources/java
added.
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-test-source (add-test-source) @ hadoop-common
---
[INFO] Test Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java
added.
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (save-version) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-common ---
[INFO] Skipped writing classpath file '/Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes/mrapp-generated-classpath'.
 No changes found.
[INFO] 
[INFO] <<< maven-source-plugin:2.1.2:jar (default) @ hadoop-common <<<
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar (default) @ hadoop-common ---
[INFO] org already added, skipping
[INFO] org/apache already added, skipping
[INFO] org/apache/hadoop already added, skipping
[INFO] org/apache/hadoop/ha already added, skipping
[INFO] org/apache/hadoop/ipc already added, skipping
[INFO] org/apache/hadoop/ipc/protobuf already added, skipping
[INFO] Building jar: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-sources.jar
[INFO] org already added, skipping
[INFO] org/apache already added, skipping
[INFO] org/apache/hadoop already added, skipping
[INFO] org/apache/hadoop/ha already added, skipping
[INFO] org/apache/hadoop/ipc already added, skipping
[INFO] org/apache/hadoop/ipc/protobuf already added, skipping
[INFO] 
[INFO] >>> maven-source-plugin:2.1.2:test-jar (default) @ hadoop-common >>>
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ hadoop-common ---
[INFO] Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-sources/java
added.
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-test-source (add-test-source) @ hadoop-common
---
[INFO] Test Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java
added.
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (save-version) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-common ---
[INFO] Skipped writing classpath file '/Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes/mrapp-generated-classpath'.
 No changes found.
[INFO] 
[INFO] <<< maven-source-plugin:2.1.2:test-jar (default) @ hadoop-common <<<
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar (default) @ hadoop-common ---
[INFO] org already added, skipping
[INFO] org/apache already added, skipping
[INFO] org/apache/hadoop already added, skipping
[INFO] org/apache/hadoop/io already added, skipping
[INFO] org/apache/hadoop/io/serializer already added, skipping
[INFO] org/apache/hadoop/io/serializer/avro already added, skipping
[INFO] org/apache/hadoop/ipc already added, skipping
[INFO] org/apache/hadoop/ipc/protobuf already added, skipping
[INFO] org/apache/hadoop/record already added, skipping
[INFO] Building jar: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-test-sources.jar
[INFO] org already added, skipping
[INFO] org/apache already added, skipping
[INFO] org/apache/hadoop already added, skipping
[INFO] org/apache/hadoop/io already added, skipping
[INFO] org/apache/hadoop/io/serializer already added, skipping
[INFO] org/apache/hadoop/io/serializer/avro already added, skipping
[INFO] org/apache/hadoop/ipc already added, skipping
[INFO] org/apache/hadoop/ipc/protobuf already added, skipping
[INFO] org/apache/hadoop/record already added, skipping
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-common ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ hadoop-common ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-common ---
[INFO] Installing /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT.jar
to /Users/harshchouraria/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT.jar
[INFO] Installing /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/pom.xml
to /Users/harshchouraria/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT.pom
[INFO] Installing /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-tests.jar
to /Users/harshchouraria/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT-tests.jar
[INFO] Installing /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-sources.jar
to /Users/harshchouraria/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT-sources.jar
[INFO] Installing /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT-test-sources.jar
to /Users/harshchouraria/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.349s
[INFO] Finished at: Fri May 25 19:25:03 GMT+05:30 2012
[INFO] Final Memory: 23M/121M
[INFO] ------------------------------------------------------------------------
{code}

+1, committing.
                
> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException
in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871-2.patch, HADOOP-6871-3.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive
substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution
check kicks in and raises an IllegalStateException indicating that the substitution depth
is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}"
will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property
value is not specified and yet being passed into the test mechanism leading to failures due
to this limitation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message