hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amith (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
Date Fri, 13 Apr 2012 16:02:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253497#comment-13253497
] 

amith commented on HDFS-3194:
-----------------------------

Hi Suja,

I have done analysis on the reported defect and came up with a patch.

DataBlockScanner thread will scan for the new blocks for every 5 seconds(new period start),
but in the scan method of BlockPoolSliceScanner finally we are rolling the logs(irrespective
of entries in logs).

When a new period starts we are loading the number of entries from the logs and non scanned
blocks are scanned.
Since there is log roll for every 5 sec after 10 seconds the previously scanned entries are
lost!!!!

causing the block scan for the next period. Thus we can see the time difference is 15 sec
for every consecutive scan.

In the provided patch I have introduced log roll only after certain number of entries in the
logs are reached(which is similar to 20.x version HDFS behaviour), care is taken not to break
log interface which already exist.
                
> Continuous block scanning at DN side
> ------------------------------------
>
>                 Key: HDFS-3194
>                 URL: https://issues.apache.org/jira/browse/HDFS-3194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: suja s
>            Priority: Minor
>             Fix For: 3.0.0
>
>         Attachments: HDFS-3194.patch
>
>
> Block scanning interval by default should be taken as 21 days(3 weeks) and each block
scanning should happen once in 21 days.
> Here the block is being scanned continuosly.
> 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message