hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3194) DataNode block scanner is running too frequently
Date Fri, 22 Jun 2012 05:52:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399136#comment-13399136
] 

Uma Maheswara Rao G commented on HDFS-3194:
-------------------------------------------

{quote}
scanBlockPoolSlice gets called every 5 seconds by DataBlockScanner.run().
{quote} 

verification logs will track the information about prvious scan time. So, while scanning,
it will load this info from this logs and will ensure that one block scanned only after 3*weeks
time.
HDFS-3194_7.patch  addresses one potential problem. For every 5 sec run, logs are simply rolling.
for every 3 slices, all verification logs are getting cleaned as max rolled index is 2 I remember
and created new ones. So, the same block will get scanned for every 15secs again and again.
I remember, we have verification log size limit in 20.x versions. So, it will not be rolled
untill it reaches that size. That has been broken here with some other JIRAs.
Let me check your point why startNewPeriod(); is required for every 5 sec slice.

                
> DataNode block scanner is running too frequently
> ------------------------------------------------
>
>                 Key: HDFS-3194
>                 URL: https://issues.apache.org/jira/browse/HDFS-3194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: suja s
>            Assignee: amith
>         Attachments: HDFS-3194.patch, HDFS-3194_1.patch, HDFS-3194_2.patch, HDFS-3194_4.patch,
HDFS-3194_6.patch, HDFS-3194_7.patch
>
>
> Block scanning interval by default should be taken as 21 days(3 weeks) and each block
scanning should happen once in 21 days.
> Here the block is being scanned continuosly.
> 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message