hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hrishikesh Gadre (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-13799) TestEditLogTailer#testTriggersLogRollsForAllStandbyNN test fail due to missing synchronization between rollEditsRpcExecutor and tailerThread shutdown
Date Mon, 06 Aug 2018 22:00:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hrishikesh Gadre updated HDFS-13799:
------------------------------------
    Attachment: HDFS-13799-001.patch

> TestEditLogTailer#testTriggersLogRollsForAllStandbyNN test fail due to missing synchronization
between rollEditsRpcExecutor and tailerThread shutdown
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-13799
>                 URL: https://issues.apache.org/jira/browse/HDFS-13799
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 3.0.0
>            Reporter: Hrishikesh Gadre
>            Assignee: Hrishikesh Gadre
>            Priority: Minor
>         Attachments: HDFS-13799-001.patch
>
>
> TestEditLogTailer#testTriggersLogRollsForAllStandbyNN unit test is failing in our internal
environment with following error,
> {noformat}
> java.lang.AssertionError: Test resulted in an unexpected exit
> 	at org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testTriggersLogRollsForAllStandbyNN(TestEditLogTailer.java:245){noformat}
> This test failure is due to following error during shutdown of the MiniDfsCluster
> {noformat}
> 2018-07-31 21:59:27,806 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1965))
- Shutting down the Mini HDFS Cluster
> 2018-07-31 21:59:27,806 [main] FATAL hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1968))
- Test resulted in an unexpected exit
> 1: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ce1d2b6
rejected from java.util.concurrent.ThreadPoolExecutor@12263f5a[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
> 	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:265)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:380)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:397)
> 	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:482)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:393)
> Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ce1d2b6
rejected from java.util.concurrent.ThreadPoolExecutor@12263f5a[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
> 	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:681)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:351)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:411)
> 	... 4 more{noformat}
> It looks like the EditLogTailer class is not handling the shutdown correctly. Specifically
the EditLogTailer#stop() method shuts down the rollEditsRpcExecutor executor service before
setting the tailerThread#shouldRun flag. This is a race condition since the tailerThread
can try to submit a new task to this executor service which has been asked to shutdown. If
that happens, it will receive an unexpected RejectedExecutionException, resulting in a test
failure. The solution should be to properly synchronize shutdown of tailerThread with rollEditsRpcExecutor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message