hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-721) ERROR Block blk_XXX_1030 already exists in state RBW and thus cannot be created
Date Wed, 21 Oct 2009 06:41:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768113#action_12768113
] 

stack commented on HDFS-721:
----------------------------

So looking in NN log, I see that block 1030 is what block 1029 became:

{code}
2009-10-21 04:57:02,890 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_6345892463926159834_1029)
successfully to blk_6345892463926159834_1030
{code}

Looking up at application level, I don't see an exception up at the application level.  Maybe
all is working properly... the block goes on to have a long and fruitful life?

Here is rest of log from the NN:

{code}
2009-10-21 04:57:05,728 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Inconsistent
size for block blk_6345892463926159834_1030 reported from XX.XX.XX.141:51010 current size
is 3056128 reported size is 67108864
2009-10-21 04:57:05,729 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
blockMap updated: XX.XX.XX.141:51010 is added to blk_6345892463926159834_1030{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XX.XX.XX.139:51010|RBW], ReplicaUnderConstruction[XX.XX.XX.141:51010|RBW]]}
size 67108864
2009-10-21 04:57:05,733 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
blockMap updated: XX.XX.XX.139:51010 is added to blk_6345892463926159834_1030{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XX.XX.XX.139:51010|RBW], ReplicaUnderConstruction[XX.XX.XX.141:51010|RBW]]}
size 67108864
2009-10-21 04:57:07,769 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask XX.XX.XX.141:51010
to replicate blk_6345892463926159834_1030 to datanode(s) XX.XX.XX.140:51010
2009-10-21 05:05:22,268 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: PendingReplicationMonitor
timed out block blk_6345892463926159834_1030
2009-10-21 05:05:27,868 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask XX.XX.XX.139:51010
to replicate blk_6345892463926159834_1030 to datanode(s) XX.XX.XX.142:51010
2009-10-21 05:05:30,299 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock:
blockMap updated: XX.XX.XX.142:51010 is added to blk_6345892463926159834_1030 size 67108864
2009-10-21 05:06:47,804 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates:
blk_6345892463926159834_1030 to XX.XX.XX.139:51010 XX.XX.XX.141:51010 XX.XX.XX.142:51010 
2009-10-21 05:06:49,139 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask XX.XX.XX.142:51010
to delete  blk_-1883563394128357682_1221 blk_-1418192880479762585_1363 blk_-7630118490185950160_1317
blk_-8001019548793370364_1148 blk_4870116241583532381_1218 blk_2108940364850350519_1082 blk_1036824913411181109_1091
blk_-7589777526384929149_1033 blk_8923577073812832890_1027 blk_-1109307457918706344_1034 blk_-116109986645631967_1219
blk_-1553219148041957509_1367 blk_3763728034862268069_1222 blk_7960880682534006819_1267 blk_-8530634816448964605_1224
blk_666438451332088436_1103 blk_-1943198821321042756_1010 blk_3860462222387154865_1032 blk_-2887612715667758540_1342
blk_2678719195781934773_1348 blk_7531114468370112370_1018 blk_5376641334456236592_1263 blk_5786815585048721793_1025
blk_2870673990810172323_1079 blk_-5877030662574024321_1055 blk_-4584853327269407588_1021 blk_6345892463926159834_1030
blk_8408287293691629994_1268 blk_8109921405484286254_1031 blk_7335139466236556727_1320 blk_1278349007490858566_1272
2009-10-21 05:06:52,149 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask XX.XX.XX.139:51010
to delete  blk_-1883563394128357682_1221 blk_-7630118490185950160_1317 blk_-8001019548793370364_1148
blk_4870116241583532381_1218 blk_2108940364850350519_1082 blk_1036824913411181109_1091 blk_-1919844007905419086_1019
blk_8923577073812832890_1027 blk_-116109986645631967_1219 blk_7960880682534006819_1267 blk_666438451332088436_1103
blk_2067665234645028297_1017 blk_-5406582365187584682_1322 blk_3860462222387154865_1032 blk_-6377718937631289761_1223
blk_2678719195781934773_1348 blk_-2648964128352415897_1023 blk_-8349995959668148332_1051 blk_-4114818907728161248_1035
blk_568068918288377484_1294 blk_-4584853327269407588_1021 blk_6345892463926159834_1030 blk_8408287293691629994_1268
blk_-4335633782362435415_1338 blk_-4179867072599502920_1265 blk_7335139466236556727_1320
2009-10-21 05:06:52,149 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask XX.XX.XX.141:51010
to delete  blk_-1418192880479762585_1363 blk_-7589777526384929149_1033 blk_-1919844007905419086_1019
blk_-1109307457918706344_1034 blk_-1553219148041957509_1367 blk_3763728034862268069_1222 blk_-8530634816448964605_1224
blk_2067665234645028297_1017 blk_-1943198821321042756_1010 blk_-5406582365187584682_1322 blk_-6377718937631289761_1223
blk_-2887612715667758540_1342 blk_-2648964128352415897_1023 blk_7531114468370112370_1018 blk_-8349995959668148332_1051
blk_5376641334456236592_1263 blk_-3674434173543740123_1147 blk_-4114818907728161248_1035 blk_5786815585048721793_1025
blk_2870673990810172323_1079 blk_-5877030662574024321_1055 blk_568068918288377484_1294 blk_6345892463926159834_1030
blk_-4335633782362435415_1338 blk_8109921405484286254_1031 blk_-4179867072599502920_1265 blk_1278349007490858566_1272
{code}

> ERROR Block blk_XXX_1030 already exists in state RBW and thus cannot be created
> -------------------------------------------------------------------------------
>
>                 Key: HDFS-721
>                 URL: https://issues.apache.org/jira/browse/HDFS-721
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: dfs.support.append=true
> Current branch-0.21 of hdfs, mapreduce, and common. Here is svn info:
> URL: https://svn.apache.org/repos/asf/hadoop/hdfs/branches/branch-0.21
> Repository Root: https://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 827883
> Node Kind: directory
> Schedule: normal
> Last Changed Author: szetszwo
> Last Changed Rev: 826906
> Last Changed Date: 2009-10-20 00:16:25 +0000 (Tue, 20 Oct 2009)
>            Reporter: stack
>
> Running some loading tests against hdfs branch-0.21 I got the following:
> {code}
> 2009-10-21 04:57:10,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
block blk_6345892463926159834_1030 src: /XX.XX.XX.141:53112 dest: /XX.XX.XX.140:51010
> 2009-10-21 04:57:10,771 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_6345892463926159834_1030 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException:
Block blk_6345892463926159834_1030 already exists in state RBW and thus cannot be created.
> 2009-10-21 04:57:10,771 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(XX.XX.XX.140:51010,
storageID=DS-1292310101-XX.XX.XX.140-51010-1256100924816, infoPort=51075, ipcPort=51020):DataXceiver
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block blk_6345892463926159834_1030
already exists in state RBW and thus cannot be created.
>     at org.apache.hadoop.hdfs.server.datanode.FSDataset.createTemporary(FSDataset.java:1324)
>     at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:258)
>     at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:382)
>     at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:323)
>     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111)
>     at java.lang.Thread.run(Thread.java:619)
> {code}
> On the sender side:
> {code}
> 2009-10-21 04:57:10,740 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(XX.XX.XX.141:51010,
storageID=DS-1870884070-XX.XX.XX.141-51010-1256100925196, infoPort=51075, ipcPort=51020) Starting
thread to transfer block blk_6345892463926159834_1030 to XX.XX.XX.140:51010
> 2009-10-21 04:57:10,770 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(XX.XX.XX.141:51010,
storageID=DS-1870884070-XX.XX.XX.141-51010-1256100925196, infoPort=51075, ipcPort=51020):Failed
to transfer blk_6345892463926159834_1030 to XX.XX.XX.140:51010 got java.net.SocketException:
Original Exception : java.io.IOException: Connection reset by peer
>     at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>     at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
>     at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
>     at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:199)
>     at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:346)
>     at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:434)
>     at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1262)
>     at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Connection reset by peer
>     ... 8 more
> {code}
> The block sequence number, 1030, is one more than that in issue HDFS-720 (same test run
but about 8 seconds between errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message