hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Behroz Sikander (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6360) MiniDFSCluster can cause unexpected side effects due to sharing of config
Date Mon, 07 Jan 2019 11:58:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735717#comment-16735717

Behroz Sikander commented on HDFS-6360:

Are there any plans to fix this issue?
We are also facing unexpected behavior of testcases.

> MiniDFSCluster can cause unexpected side effects due to sharing of config
> -------------------------------------------------------------------------
>                 Key: HDFS-6360
>                 URL: https://issues.apache.org/jira/browse/HDFS-6360
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Priority: Major
> As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can result in
unexpected results and falsely failing or passing unit tests.
> Since a {{Configuration}} object is shared for all namenode startups, the modified conf
object during a NN startup is passed to the next NN startup.  The effect of the modified conf
propagation and subsequent modifications is different depending on whether it is a single
NN cluster, HA cluster or federation cluster.
> It also depends on what test cases are doing with the config. For example, MiniDFSCluster#getConfiguration(int)
returns the saved conf for the specified NN, but that is not actually the conf object used
by the NN. It just contained the same content one time in the past and it is not guaranteed
to be that way.
> Restarting the same NN can also cause unexpected results. The new NN will switch to the
conf that was cloned & saved AFTER the last startup.  The new NN will start with a changed
config intentionally or unintentionally.  The config variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}}
will be implicitly set differently than the initial condition.  Some test cases rely on this
and others occasionally break because of this.
> In summary,
> * MiniDFSCluster does not properly isolate configs.
> * Many test cases happen to work most of times. Correcting MiniDFSCluster causes mass
breakages of test cases and requires fixing them.
> * Many test cases rely on broken behavior and might pass when they should have actually
> We need to
> * Make MiniDFSCluster behave in a consistent way
> * Provide proper methods and documentation for the correct usage of MiniDFSCluster
> * Fix the unit tests that will be broken after improving MiniDFSCluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message