Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 23 Aug 2012 06:43:43 +1100 (NCT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <1416399259.2069.1345664623083.JavaMail.jiratomcat@arcas>
In-Reply-To: <1445695877.36856.1345579838292.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (HDFS-3828) Block Scanner rescans blocks too
 frequently
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439793#comment-13439793 ] 

Eli Collins commented on HDFS-3828:
-----------------------------------

Thanks for filing/fixing Andy and Amith/Uma for identifying this!
 
It looks like the following code in BlockPoolSliceScanner#scan was intended to regular the interval, but per HDFS-3194 doesn't work because the call to cleanUp() rolls the log.  What's the advantage of this fix vs just making cleanUp (which should be renamed) only roll the log at the appropriate interval?  If we go with the approch in your patch then we should make this code a precondition and always start a new period as there's no way we should be in scan w/o a new period given you're new check.

{code}
if ( now >= (currentPeriodStart + scanPeriod)) {
  startNewPeriod();
}
{code}

- Please update the jira description with a description of the bug
- Let's rename "cleanUp" to something like "rollVerificationLog"
- If the scanner scans exactly once shouldn't scansLastRun be 0 after this first run? Ie getBlocksScannedInLastRun shouldn't always return 1 right?
- How about defining a constant (with package protection) to use in DataBlockScanner's sleep call and your test.
- waitAndScanBlocks should catch InterruptedException right after the sleep (and fail()) rather than propogate it outside the test.
- Not your change but mind fixing the spelling mistake / gross "//Sleep everytime except in the first interation." comment

                
> Block Scanner rescans blocks too frequently
> -------------------------------------------
>
>                 Key: HDFS-3828
>                 URL: https://issues.apache.org/jira/browse/HDFS-3828
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.23.0, 2.0.0-alpha
>            Reporter: Andy Isaacson
>            Assignee: Andy Isaacson
>         Attachments: hdfs3828.txt
>
>
> Branching off from HDFS-3194:
> Andy said:
> {quote}
> There is still an issue here even with the patch from hdfs-3194-1.txt. Running on a cluster with just one block we now see
> {noformat}
> 2012-08-16 15:59:57,884 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:07,904 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> 2012-08-16 16:00:17,925 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
> {noformat}
> {quote}
> Uma said:
> bq. Latest patch from Amith should address this issue. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira