hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amith (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3512) Delay in scanning blocks at DN side when there are huge number of blocks
Date Thu, 07 Jun 2012 14:42:27 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

amith updated HDFS-3512:

    Attachment: HDFS-3512.patch

Currently we are adding some random time when the block is getting added into blockInfoSet,
this randomization will be for a very short period of time i.e, till the block is scanned
for the first time, after first scan it will update the exact time of scan so I thought of
removing the random and verified the block scanning for 75K blocks which worked fine.

I tried to write a test for the same but couldn't since I can't determine the random number
generated and added.

@Suja can u just apply the patch and test the same
> Delay in scanning blocks at DN side when there are huge number of blocks
> ------------------------------------------------------------------------
>                 Key: HDFS-3512
>                 URL: https://issues.apache.org/jira/browse/HDFS-3512
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 2.0.1-alpha
>            Reporter: suja s
>            Assignee: amith
>         Attachments: HDFS-3512.patch
> Block scanner maintains the full list of blocks at DN side in a map and there is no differentiation
between the blocks which are already scanned and the ones not scanend. For every check (ie
every 5 secs) it will pick one block and scan. There are chances that it chooses a block which
is already scanned which leads to further delay in scanning of blcoks which are yet to be

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message