hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-6360) MiniDFSCluster can cause unexpected side effects due to sharing of config
Date Sat, 10 May 2014 22:11:14 GMT
Kihwal Lee created HDFS-6360:

             Summary: MiniDFSCluster can cause unexpected side effects due to sharing of config
                 Key: HDFS-6360
                 URL: https://issues.apache.org/jira/browse/HDFS-6360
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Kihwal Lee

As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can result in unexpected
results and falsely failing or passing unit tests.

Since a {{Configuration}} object is shared for all namenode startups, the modified conf object
during a NN startup is passed to the next NN startup.  The effect of the modified conf propagation
and subsequent modifications is different depending on whether it is a single NN cluster,
HA cluster or federation cluster.

It also depends on what test cases are doing with the config. For example, MiniDFSCluster#getConfiguration(int)
returns the saved conf for the specified NN, but that is not actually the conf object used
by the NN. It just contained the same content one time in the past and it is not guaranteed
to be that way.

Restarting the same NN can also cause unexpected results. The new NN will switch to the conf
that was cloned & saved AFTER the last startup.  The new NN will start with a changed
config intentionally or unintentionally.  The config variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}}
will be implicitly set differently than the initial condition.  Some test cases rely on this
and others occasionally break because of this.

In summary,
* MiniDFSCluster does not properly isolate configs.
* Many test cases happen to work most of times. Correcting MiniDFSCluster causes mass breakages
of test cases and requires fixing them.
* Many test cases rely on broken behavior and might pass when they should have actually failed.

This message was sent by Atlassian JIRA

View raw message