hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Isaacson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3194) DataNode block scanner is running too frequently
Date Thu, 16 Aug 2012 23:03:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436398#comment-13436398
] 

Andy Isaacson commented on HDFS-3194:
-------------------------------------

bq. Just re-opned this JIRA to check the above mentioned scenario.

There is still an issue here even with the patch from hdfs-3194-1.txt.  Running on a cluster
with just one block we now see
{noformat}
2012-08-16 15:59:57,884 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
2012-08-16 16:00:07,904 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
2012-08-16 16:00:17,925 INFO  datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391))
- Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915
{noformat}
the BP scanner is re-verifying the same replica every 10 seconds.

We're not restarting a new period though, fetching blockScannerReport?listblocks shows a reasonable
amount of "Time left in cur period" although "Progress in this period" is badly wrong:
{noformat}
Block report for block pool: BP-2101131164-172.29.122.91-1337906886255
blk_7919273167187535506_4915 : status : ok     type : local  scan time : 1345158298637   2012-08-16
16:04:58,637

Total Blocks                 :      1
Verified in last hour        :      1
Verified in last day         :      1
Verified in last week        :      1
Verified in last four weeks  :      1
Verified in SCAN_PERIOD      :      1
Not yet verified             :      0
Verified since restart       :     31
Scans since restart          :     31
Scan errors since restart    :      0
Transient scan errors        :      0
Current scan rate limit KBps :   1024
Progress this period         :   3200%
Time left in cur period      :  99.98%
{noformat}
                
> DataNode block scanner is running too frequently
> ------------------------------------------------
>
>                 Key: HDFS-3194
>                 URL: https://issues.apache.org/jira/browse/HDFS-3194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: suja s
>            Assignee: Andy Isaacson
>             Fix For: 2.2.0-alpha
>
>         Attachments: HDFS-3194_1.patch, hdfs-3194-1.txt, HDFS-3194_2.patch, HDFS-3194_4.patch,
HDFS-3194_6.patch, HDFS-3194_7.patch, HDFS-3194.patch
>
>
> Block scanning interval by default should be taken as 21 days(3 weeks) and each block
scanning should happen once in 21 days.
> Here the block is being scanned continuosly.
> 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message