Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 13 Apr 2012 16:02:03 +0000 (UTC)
From: "amith (Commented) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1190980068.22513.1334332923806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1942965881.10866.1333532063781.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253496#comment-13253496 ] 

amith commented on HDFS-3194:
-----------------------------

Hi Suja,

I have done analysis on the reported defect and came up with a patch.

DataBlockScanner thread will scan for the new blocks for every 5 seconds(new period start), but in the scan method of BlockPoolSliceScanner finally we are rolling the logs(irrespective of entries in logs).

When a new period starts we are loading the number of entries from the logs and non scanned blocks are scanned.
Since there is log roll for every 5 sec after 10 seconds the previously scanned entries are lost!!!!

causing the block scan for the next period. Thus we can see the time difference is 15 sec for every consecutive scan.

In the provided patch I have introduced log roll only after certain number of entries in the logs are reached(which is similar to 20.x version HDFS behaviour), care is taken not to break log interface which already exist.
                
> Continuous block scanning at DN side
> ------------------------------------
>
>                 Key: HDFS-3194
>                 URL: https://issues.apache.org/jira/browse/HDFS-3194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: suja s
>            Priority: Minor
>             Fix For: 3.0.0
>
>         Attachments: HDFS-3194.patch
>
>
> Block scanning interval by default should be taken as 21 days(3 weeks) and each block scanning should happen once in 21 days.
> Here the block is being scanned continuosly.
> 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira