hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10592) Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
Date Mon, 04 Jul 2016 17:40:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361632#comment-15361632
] 

Rakesh R commented on HDFS-10592:
---------------------------------

As per the analysis the test case failure is due to the {{ConcurrentModificationException}}
while iterating over the mockito stubs.

Following are the two possible concurrent operations which results in the above exception.

operation-1) NameNodeResourceMonitor is running and invoked {{FSNamesystem.checkAvailableResources}}
operation-2) At the same time, adding the answer {{Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false);}}
to the {{InvocationContainerImpl#stubbed}} linked list data structure.

Reference log:
{code}
2016-07-04 12:00:05,657 [DataNode: [[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/,
[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]]
 heartbeating to localhost/127.0.0.1:46553] INFO  datanode.DataNode (BPOfferService.java:processCommandFromActive(696))
- Got finalize command for block pool BP-1174120436-172.17.0.2-1467633603414
2016-07-04 12:00:05,757 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543))
- Cluster is active
2016-07-04 12:00:05,760 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1]
ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in NameNodeResourceMonitor:

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71)
	at org.mockito.internal.MockHandler.handle(MockHandler.java:93)
	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791)
	at java.lang.Thread.run(Thread.java:745)
2016-07-04 12:01:05,772 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907))
- Shutting down the Mini HDFS Cluster
2016-07-04 12:01:05,773 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(1947))
- Shutting down DataNode 0
{code}

org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not thread safe.
{code}
org.mockito.internal.stubbing.InvocationContainerImpl.java
    public StubbedInvocationMatcher findAnswerFor(Invocation invocation) {
        for (StubbedInvocationMatcher s : stubbed) {
            if (s.matches(invocation)) {
                s.markStubUsed(invocation);
                invocation.markStubbed(new StubInfo(s));
                return s;
            }
        }

        return null;
    }
{code}

*Proposed Fix:*
Instead of using {{Mockito}}, it is pretty simple to use {{NameNodeResourceCheckerStub}} stubbing
and sets the resource availability explicitly. I will attach a patch soon.

> Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10592
>                 URL: https://issues.apache.org/jira/browse/HDFS-10592
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>
> This jira is to fix the {{TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning}}
test case failure.
> Reference [Build_15973|https://builds.apache.org/job/PreCommit-HDFS-Build/15973/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNameNodeResourceChecker/testCheckThatNameNodeResourceMonitorIsRunning/]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message