hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6532) Intermittent test failure org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
Date Thu, 04 Aug 2016 16:37:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408077#comment-15408077
] 

Kihwal Lee commented on HDFS-6532:
----------------------------------

Still sometimes failing in trunk the same way.

When it is working, {{close()}} should fail.
{noformat}
2016-08-04 11:10:38,293 [Thread-0] INFO  hdfs.DFSClient
 (TestCrcCorruption.java:testCorruptionDuringWrt(140)) - Got expected exception
java.io.IOException: Failing write. Tried pipeline recovery 5 times without success.
        at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1128)
        at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:552)
{noformat}

In the failed case, pipeline recovery only happened twice. {{DataStreamer}} usually directly
notices a problem or {{ResponseProcessor}} hints it. It looks like the datanode thread tried
to terminate, but the connection was not closed.

> Intermittent test failure org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6532
>                 URL: https://issues.apache.org/jira/browse/HDFS-6532
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, hdfs-client
>    Affects Versions: 2.4.0
>            Reporter: Yongjun Zhang
>
> Per https://builds.apache.org/job/Hadoop-Hdfs-trunk/1774/testReport, we had the following
failure. Local rerun is successful
> {code}
> Regression
> org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
> Failing for the past 1 build (Since Failed#1774 )
> Took 50 sec.
> Error Message
> test timed out after 50000 milliseconds
> Stacktrace
> java.lang.Exception: test timed out after 50000 milliseconds
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024)
> 	at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008)
> 	at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> 	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98)
> 	at org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133)
> {code}
> See relevant exceptions in log
> {code}
> 2014-06-14 11:56:15,283 WARN  datanode.DataNode (BlockReceiver.java:verifyChunks(404))
- Checksum error in block BP-1675558312-67.195.138.30-1402746971712:blk_1073741825_1001 from
/127.0.0.1:41708
> org.apache.hadoop.fs.ChecksumException: Checksum error: DFSClient_NONMAPREDUCE_-1139495951_8
at 64512 exp: 1379611785 got: -12163112
> 	at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:353)
> 	at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:284)
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:402)
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:537)
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> 	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> 	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> 	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> 	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> 	at java.lang.Thread.run(Thread.java:662)
> 2014-06-14 11:56:15,285 WARN  datanode.DataNode (BlockReceiver.java:run(1207)) - IOException
in BlockReceiver.run(): 
> java.io.IOException: Shutting down writer and responder due to a checksum error in received
data. The error response has been sent upstream.
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1352)
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1278)
> 	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1199)
> 	at java.lang.Thread.run(Thread.java:662)
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message