hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-6114) Block Scan log rolling will never happen if blocks written continuously leading to huge size of dncp_block_verification.log.curr
Date Tue, 18 Mar 2014 09:05:44 GMT
Vinayakumar B created HDFS-6114:
-----------------------------------

             Summary: Block Scan log rolling will never happen if blocks written continuously
leading to huge size of dncp_block_verification.log.curr
                 Key: HDFS-6114
                 URL: https://issues.apache.org/jira/browse/HDFS-6114
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.3.0
            Reporter: Vinayakumar B
            Assignee: Vinayakumar B
            Priority: Critical


1. {{BlockPoolSliceScanner#scan()}} will not return until all the blocks are scanned. 
2. If the blocks (with size in several MBs) to datanode are written continuously 
then one iteration of {{BlockPoolSliceScanner#scan()}} will be continously scanning the blocks
3. These blocks will be deleted after some time (enough to get block scanned)
4. As Block Scanning is throttled, So verification of all blocks will take so much time.
5. Rolling will never happen, so even though the total number of blocks in datanode doesn't
increases, entries ( which contains stale entries of deleted blocks) in *dncp_block_verification.log.curr*
continuously increases leading to huge size.

In one of our env, it grown more than 1TB where total number of blocks were only ~45k.






--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message