hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4274) BlockPoolSliceScanner does not close verification log during shutdown
Date Wed, 12 Dec 2012 00:51:21 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HDFS-4274:

    Attachment: HDFS-4274.2.patch

Here is the new version of the patch:

# During {{DataNode}} shutdown, change {{DataBlockScanner#run}} to shut down every allocated
{{BlockPoolSliceScanner}}, which in turn closes the verification logs.  This part is the same
as the earlier patch.
# {{DataBlockScanner#shutdown}} also joins to the background blockScannerThread.  There is
a slightly tricky bit in here about avoiding a deadlock.  I explained it in comments.  This
is necessary to make shutdown synchronous, so that the caller knows deterministically that
shutdown has completed.  In the case of {{MiniDFSCluster}}, it means that it is safe to delete
the working directories, because nothing will be holding locks.
# Change {{BlockPoolSliceScanner#scan}} to use {{Thread#isInterrupted}} instead of {{Thread#interrupted}}.
 {{BlockPoolSliceScanner#scan}} runs within the blockScannerThread, which depends on seeing
the interrupted status of the thread to know when to exit.  (See the while loop condition
in {{DataBlockScanner#run}}.)  When {{BlockPoolSliceScanner#scan}} was calling {{Thread#interrupted}},
it would clear the current thread's interrupted status, so that the while loop in {{DataBlockScanner#run}}
would never see it, and therefore would never exit.  This was occasionally causing tests to
enter an infinite loop on Windows.

With this patch in place, I am consistently seeing good test results on Mac and Windows, with
no tests blocked on a prior {{MiniDFSCluster}} holding a lock on a verification log.
> BlockPoolSliceScanner does not close verification log during shutdown
> ---------------------------------------------------------------------
>                 Key: HDFS-4274
>                 URL: https://issues.apache.org/jira/browse/HDFS-4274
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.0.0, trunk-win
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4274.1.patch, HDFS-4274.2.patch
> {{BlockPoolSliceScanner}} holds open a handle to a verification log.  This file is not
getting closed during process shutdown.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message