hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5809) BlockPoolSliceScanner and high speed hdfs appending make datanode to drop into infinite loop
Date Tue, 15 Jul 2014 16:53:05 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Colin Patrick McCabe updated HDFS-5809:
---------------------------------------

    Description: 
{{BlockPoolSliceScanner#scan}} contains a "while" loop that continues to verify (i.e. scan)
blocks until the {{blockInfoSet}} is empty (or some other conditions like a timeout have occurred.)
 In order to do this, it calls {{BlockPoolSliceScanner#verifyFirstBlock}}.  This is intended
to grab the first block in the {{blockInfoSet}}, verify it, and remove it from that set. 
({{blockInfoSet}} is sorted by last scan time.) Unfortunately, if we hit a certain bug in
{{updateScanStatus}}, the block may never be removed from {{blockInfoSet}}.  When this happens,
we keep rescanning the exact same block until the timeout hits.

The bug is triggered when a block winds up in {{blockInfoSet}} but not in {{blockMap}}.  You
can see it clearly in this code:
{code}
  private synchronized void updateScanStatus(Block block,                      
                                             ScanType type,
                                             boolean scanOk) {                 
    BlockScanInfo info = blockMap.get(block);
                                                                               
    if ( info != null ) {
      delBlockInfo(info);
    } else {                                                                   
      // It might already be removed. Thats ok, it will be caught next time.   
      info = new BlockScanInfo(block);                                         
    }   
{code}

If {{info == null}}, we never call {{delBlockInfo}}, the function which is intended to remove
the {{blockInfoSet}} entry.

Luckily, there is a simple fix here... the variable that {{updateScanStatus}} is being passed
is actually a BlockInfo object, so we can simply call {{delBlockInfo}} on it directly, without
doing a lookup in the {{blockMap}}.  This is both faster and more robust.

  was:
Hello, everyone.

When hadoop cluster starts, BlockPoolSliceScanner start scanning the blocks in my cluster.
Then, randomly one datanode drop into infinite loop as the log show, and finally all datanodes
drop into infinite loop.
Every datanode just verify fail by one block. 
When i check the fail block like this : hadoop fsck / -files -blocks | grep blk_1223474551535936089_4702249,
no hdfs file contains the block.

It seems that in while block of BlockPoolSliceScanner's scan method drop into infinite loop
.
BlockPoolSliceScanner: 650

while (datanode.shouldRun
&& !datanode.blockScanner.blockScannerThread.isInterrupted()
&& datanode.isBPServiceAlive(blockPoolId)) { ....

The log finally printed in method verifyBlock(BlockPoolSliceScanner:453).

Please excuse my poor English.
-------------------------------------------------------------------------------------------------------------------------------------------------
LOG: 
2014-01-21 18:36:50,582 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification failed for BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634
- may be due to race with write
2014-01-21 18:36:50,582 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification failed for BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634
- may be due to race with write
2014-01-21 18:36:50,582 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification failed for BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634
- may be due to race with write


> BlockPoolSliceScanner and high speed hdfs appending make datanode to drop into infinite
loop
> --------------------------------------------------------------------------------------------
>
>                 Key: HDFS-5809
>                 URL: https://issues.apache.org/jira/browse/HDFS-5809
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.0.0-alpha
>         Environment: jdk1.6, centos6.4, 2.0.0-cdh4.5.0
>            Reporter: ikweesung
>            Assignee: Colin Patrick McCabe
>            Priority: Critical
>              Labels: blockpoolslicescanner, datanode, infinite-loop
>         Attachments: HDFS-5809.001.patch
>
>
> {{BlockPoolSliceScanner#scan}} contains a "while" loop that continues to verify (i.e.
scan) blocks until the {{blockInfoSet}} is empty (or some other conditions like a timeout
have occurred.)  In order to do this, it calls {{BlockPoolSliceScanner#verifyFirstBlock}}.
 This is intended to grab the first block in the {{blockInfoSet}}, verify it, and remove it
from that set.  ({{blockInfoSet}} is sorted by last scan time.) Unfortunately, if we hit a
certain bug in {{updateScanStatus}}, the block may never be removed from {{blockInfoSet}}.
 When this happens, we keep rescanning the exact same block until the timeout hits.
> The bug is triggered when a block winds up in {{blockInfoSet}} but not in {{blockMap}}.
 You can see it clearly in this code:
> {code}
>   private synchronized void updateScanStatus(Block block,                      
>                                              ScanType type,
>                                              boolean scanOk) {                 
>     BlockScanInfo info = blockMap.get(block);
>                                                                                
>     if ( info != null ) {
>       delBlockInfo(info);
>     } else {                                                                   
>       // It might already be removed. Thats ok, it will be caught next time.   
>       info = new BlockScanInfo(block);                                         
>     }   
> {code}
> If {{info == null}}, we never call {{delBlockInfo}}, the function which is intended to
remove the {{blockInfoSet}} entry.
> Luckily, there is a simple fix here... the variable that {{updateScanStatus}} is being
passed is actually a BlockInfo object, so we can simply call {{delBlockInfo}} on it directly,
without doing a lookup in the {{blockMap}}.  This is both faster and more robust.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message