hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drake민영근 <drake....@nexr.com>
Subject HDFS Datanode Blockreport includes invalid block.
Date Thu, 01 Oct 2015 04:56:46 GMT
Greetings, All

Recently, I met the worst case of HDFS, the missing block. The
timeline(log) of namenode is below:

initial state: Datanode A(or B or C. not confirm yet) and 192.168.100.90
contains block_1.

(Datanode A, B, C crash w/ hardware fault)
07:33:03: ask replication block_1 from 192.168.100.90 to 192.168.100.210
07:34:14: updatedBlockmap addStoredBlock 192.168.100.210 with block_1
...
(Datanode A, B, C recover)
18:11:36: block_1 of 192.168.100.210 is invalidate
18:11:45: block_1 of 192.168.100.210 is deleted
...
19:43:03: updatedBlockmap addStoredBlock 192.168.100.210 with block_1
19:43:03: block_1 of 192.168.100.90 is invalidate
19:43:04: block_1 of 192.168.100.90 is deleted
...
(Datanode A, B, C crash w/ hardware fault AGAIN)
00:27:21: ask replication block_1 from 192.168.100.210 to 192.168.100.82
07:34:14: Error cause block_1 of 192.168.100.210 is invalid

At 19:43:03, Datanode 192.168.100.210 send the block report to namenode. I
guess 192.168.100.210's block report contains worng, in this case invalid,
block.

Anyone seen this problem ?

Sorry for log format. I cannot get the full logs.

Thanks.

Drake 민영근 Ph.D
kt NexR

Mime
View raw message