Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 4 Jul 2016 17:40:10 +0000 (UTC)
From: "Rakesh R (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12986648.1467653049000.20688.1467654010967@Atlassian.JIRA>
In-Reply-To: <JIRA.12986648.1467653049000@Atlassian.JIRA>
References: <JIRA.12986648.1467653049000@Atlassian.JIRA> <JIRA.12986648.1467653049126@arcas>
Subject: [jira] [Commented] (HDFS-10592) Fix intermittent test failure of
 TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 04 Jul 2016 17:40:13 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361632#comment-15361632 ] 

Rakesh R commented on HDFS-10592:
---------------------------------

As per the analysis the test case failure is due to the {{ConcurrentModificationException}} while iterating over the mockito stubs.

Following are the two possible concurrent operations which results in the above exception.

operation-1) NameNodeResourceMonitor is running and invoked {{FSNamesystem.checkAvailableResources}}
operation-2) At the same time, adding the answer {{Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false);}} to the {{InvocationContainerImpl#stubbed}} linked list data structure.

Reference log:
{code}
2016-07-04 12:00:05,657 [DataNode: [[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/, [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:46553] INFO  datanode.DataNode (BPOfferService.java:processCommandFromActive(696)) - Got finalize command for block pool BP-1174120436-172.17.0.2-1467633603414
2016-07-04 12:00:05,757 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active
2016-07-04 12:00:05,760 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1] ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in NameNodeResourceMonitor: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71)
	at org.mockito.internal.MockHandler.handle(MockHandler.java:93)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791)
	at java.lang.Thread.run(Thread.java:745)
2016-07-04 12:01:05,772 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster
2016-07-04 12:01:05,773 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(1947)) - Shutting down DataNode 0
{code}

org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not thread safe.
{code}
org.mockito.internal.stubbing.InvocationContainerImpl.java
    public StubbedInvocationMatcher findAnswerFor(Invocation invocation) {
        for (StubbedInvocationMatcher s : stubbed) {
            if (s.matches(invocation)) {
                s.markStubUsed(invocation);
                invocation.markStubbed(new StubInfo(s));
                return s;
            }
        }

        return null;
    }
{code}

*Proposed Fix:*
Instead of using {{Mockito}}, it is pretty simple to use {{NameNodeResourceCheckerStub}} stubbing and sets the resource availability explicitly. I will attach a patch soon.

> Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10592
>                 URL: https://issues.apache.org/jira/browse/HDFS-10592
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>
> This jira is to fix the {{TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning}} test case failure.
> Reference [Build_15973|https://builds.apache.org/job/PreCommit-HDFS-Build/15973/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNameNodeResourceChecker/testCheckThatNameNodeResourceMonitorIsRunning/]


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org