hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3828) Block Scanner rescans blocks too frequently
Date Wed, 22 Aug 2012 19:43:43 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439793#comment-13439793

Eli Collins commented on HDFS-3828:

Thanks for filing/fixing Andy and Amith/Uma for identifying this!
It looks like the following code in BlockPoolSliceScanner#scan was intended to regular the
interval, but per HDFS-3194 doesn't work because the call to cleanUp() rolls the log.  What's
the advantage of this fix vs just making cleanUp (which should be renamed) only roll the log
at the appropriate interval?  If we go with the approch in your patch then we should make
this code a precondition and always start a new period as there's no way we should be in scan
w/o a new period given you're new check.

if ( now >= (currentPeriodStart + scanPeriod)) {

- Please update the jira description with a description of the bug
- Let's rename "cleanUp" to something like "rollVerificationLog"
- If the scanner scans exactly once shouldn't scansLastRun be 0 after this first run? Ie getBlocksScannedInLastRun
shouldn't always return 1 right?
- How about defining a constant (with package protection) to use in DataBlockScanner's sleep
call and your test.
- waitAndScanBlocks should catch InterruptedException right after the sleep (and fail()) rather
than propogate it outside the test.
- Not your change but mind fixing the spelling mistake / gross "//Sleep everytime except in
the first interation." comment

> Block Scanner rescans blocks too frequently
> -------------------------------------------
>                 Key: HDFS-3828
>                 URL: https://issues.apache.org/jira/browse/HDFS-3828
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.23.0, 2.0.0-alpha
>            Reporter: Andy Isaacson
>            Assignee: Andy Isaacson
>         Attachments: hdfs3828.txt
> Branching off from HDFS-3194:
> Andy said:
> {quote}
> There is still an issue here even with the patch from hdfs-3194-1.txt. Running on a cluster
with just one block we now see
> {noformat}
> 2012-08-16 15:59:57,884 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-
> 2012-08-16 16:00:07,904 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-
> 2012-08-16 16:00:17,925 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-
> {noformat}
> {quote}
> Uma said:
> bq. Latest patch from Amith should address this issue. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message