hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amith (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3194) DataNode block scanner is running too frequently
Date Thu, 31 May 2012 05:34:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286337#comment-13286337
] 

amith commented on HDFS-3194:
-----------------------------

Hi Eli 
Thanks for looking into this issue

This issue may be induced due to some federation related changes (I don't know which)

{quote}I think the current behavior (rolling the log on each period) is intended. I think
the bug is that we're starting a new period way too frequently (every 5 seconds instead of
every three weeks by default). DataBlockScanner#run calls scanBlockPoolSlice every 5 seconds{quote}

I feel that code will be modified some thing like 
{code}
//Sleep everytime except in the first interation.
      if (!firstRun) {
        try {
          Thread.sleep(21*24*3600);
        } catch (InterruptedException ex) {
          // Interrupt itself again to set the interrupt status
          blockScannerThread.interrupt();
          continue;
        }
{code}

If I change the code like this then there is a possibly we will not be able to do incremental
block scan 
eg: I will write 10 blocks @time  14:53:46
now block scanner has scanned the blocks 
I wrote another set of blocks in 15:53:46 then this blocks will not be scanned by block scanner
for next 3 weeks.

Now in my change this is same as Hadoop 0.20.X behavior (Even new period is started logs are
rolled only after reaching certain limit). 
where logs will be rolled only after certain number of blocks are reached in the logs.

Please correct me if I am wrong :)

Thanks 
Amith

                
> DataNode block scanner is running too frequently
> ------------------------------------------------
>
>                 Key: HDFS-3194
>                 URL: https://issues.apache.org/jira/browse/HDFS-3194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: suja s
>            Assignee: amith
>         Attachments: HDFS-3194.patch, HDFS-3194_1.patch, HDFS-3194_2.patch, HDFS-3194_4.patch,
HDFS-3194_6.patch
>
>
> Block scanning interval by default should be taken as 21 days(3 weeks) and each block
scanning should happen once in 21 days.
> Here the block is being scanned continuosly.
> 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003
> 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message