hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-3584) Blocks are getting marked as corrupt with append operation under high load.
Date Mon, 02 Jul 2012 06:24:44 GMT
Brahma Reddy Battula created HDFS-3584:
------------------------------------------

             Summary: Blocks are getting marked as corrupt with append operation under high
load.
                 Key: HDFS-3584
                 URL: https://issues.apache.org/jira/browse/HDFS-3584
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 2.0.1-alpha
            Reporter: Brahma Reddy Battula


Scenario:
========= 
1. There are 2 clients cli1 and cli2 cli1 write a file F1 and not closed
2. The cli2 will call append on unclosed file and triggers a leaserecovery
3. Cli1 is closed
4. Lease recovery is completed and with updated GS in DN and got BlockReport since there is
a mismatch in GS the block got corrupted
5. Now we got a CommitBlockSync this will also fail since the File is already closed by cli1
and state in NN is Finalized

 *please check following log for blk_-7909104799008701972* 



{noformat}
2012-06-25 13:48:59,603 INFO org.apache.hadoop.hdfs.StateChange: BLOCK NameSystem.addToCorruptReplicasMap:
blk_-7909104799008701972 added as corrupt on ****DN1:50010 by /****DN1 because block is COMPLETE
and reported genstamp 96470 does not match genstamp in block map 96309
2012-06-25 13:48:59,604 DEBUG org.apache.hadoop.hdfs.StateChange: UnderReplicationBlocks.update
blk_-7909104799008701972_96309 curReplicas 2 curExpectedReplicas 3 oldReplicas 3 oldExpectedReplicas
 3 curPri  2 oldPri  3
2012-06-25 13:48:59,604 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.UnderReplicationBlock.update:blk_-7909104799008701972_96309
has only 2 replicas and needs 3 replicas so is added to neededReplications at priority level
2
2012-06-25 13:48:59,604 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* block RECEIVED_BLOCK:
blk_-7909104799008701972_96470 is received from DatanodeRegistration(****DN1, storageID=DS-1986831640-****DN1-50010-1340363042399,
infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-fdfc6cef-05b1-4900-b5f9-cc275dfd343c;nsid=415242063;c=0)
2012-06-25 13:48:59,607 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_-7909104799008701972_96470 on ****DN2:50010 size 524288 replicaState =
FINALIZED
2012-06-25 13:48:59,608 INFO org.apache.hadoop.hdfs.StateChange: BLOCK NameSystem.addToCorruptReplicasMap:
blk_-7909104799008701972 added as corrupt on ****DN2:50010 by /****DN2 because block is COMPLETE
and reported genstamp 96470 does not match genstamp in block map 96309
2012-06-25 13:48:59,608 DEBUG org.apache.hadoop.hdfs.StateChange: UnderReplicationBlocks.update
blk_-7909104799008701972_96309 curReplicas 1 curExpectedReplicas 3 oldReplicas 2 oldExpectedReplicas
 3 curPri  0 oldPri  2
2012-06-25 13:48:59,609 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.UnderReplicationBlock.remove:
Removing block blk_-7909104799008701972_96309 from priority queue 2
2012-06-25 13:48:59,609 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.UnderReplicationBlock.update:blk_-7909104799008701972_96309
has only 1 replicas and needs 3 replicas so is added to neededReplications at priority level
0
2012-06-25 13:48:59,610 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* block RECEIVED_BLOCK:
blk_-7909104799008701972_96470 is received from DatanodeRegistration(****DN2, storageID=DS-485536663-****DN2-50010-1340362102909,
infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-fdfc6cef-05b1-4900-b5f9-cc275dfd343c;nsid=415242063;c=0)
2012-06-25 13:48:59,678 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_-7909104799008701972_96470 on ****DN3:50010 size 524288 replicaState =
FINALIZED
2012-06-25 13:48:59,679 INFO org.apache.hadoop.hdfs.StateChange: BLOCK NameSystem.addToCorruptReplicasMap:
blk_-7909104799008701972 added as corrupt on ****DN3:50010 by /****DN3 because block is COMPLETE
and reported genstamp 96470 does not match genstamp in block map 96309
2012-06-25 13:48:59,681 DEBUG org.apache.hadoop.hdfs.StateChange: UnderReplicationBlocks.update
blk_-7909104799008701972_96309 curReplicas 0 curExpectedReplicas 3 oldReplicas 1 oldExpectedReplicas
 3 curPri  4 oldPri  0
2012-06-25 13:48:59,681 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.UnderReplicationBlock.remove:
Removing block blk_-7909104799008701972_96309 from priority queue 0
2012-06-25 13:48:59,682 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.UnderReplicationBlock.update:blk_-7909104799008701972_96309
has only 0 replicas and needs 3 replicas so is added to neededReplications at priority level
4
2012-06-25 13:48:59,682 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* block RECEIVED_BLOCK:
blk_-7909104799008701972_96470 is received from DatanodeRegistration(****DN3, storageID=DS-598160968-****DN3-50010-1340382093938,
infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-fdfc6cef-05b1-4900-b5f9-cc275dfd343c;nsid=415242063;c=0)
2012-06-25 13:48:59,683 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: commitBlockSynchronization(lastblock=BP-1988075715-****DN1-1340361925673:blk_-7909104799008701972_96309,
newgenerationstamp=96470, newlength=524288, newtargets=[****DN1:50010, ****DN2:50010, ****DN3:50010],
closeFile=true, deleteBlock=false)
2012-06-25 13:48:59,683 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:root (auth:SIMPLE) cause:java.io.IOException: Unexpected block (=BP-1988075715-****DN1-1340361925673:blk_-7909104799008701972_96309)
since the file (=append_1654216706419640) is not under construction
2012-06-25 13:48:59,685 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 8020, call
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.commitBlockSynchronization from ****DN1:15704:
error: java.io.IOException: Unexpected block (=BP-1988075715-****DN1-1340361925673:blk_-7909104799008701972_96309)
since the file (=append_1654216706419640) is not under construction
java.io.IOException: Unexpected block (=BP-1988075715-****DN1-1340361925673:blk_-7909104799008701972_96309)
since the file (=append_1654216706419640) is not under construction
2012-06-25 13:49:00,413 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Block blk_-7909104799008701972_96309 cannot be repl from any node

{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message