hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5720) DataNode always READ_BLOCK or WRITE_BLOCK
Date Mon, 06 Jan 2014 17:09:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863111#comment-13863111
] 

Suresh Srinivas commented on HDFS-5720:
---------------------------------------

For asking questions please use the mailing list. Jira is for reporting bugs.

Looking at couple of instances from the log, the issues seems to be DFSClient reading or writing
blocks could be prematurely terminated. Exceptions may not indicate any real problems. What
issues are you observing in terms of applications and jobs running on HDFS?

For now closing this as Invalid.

> DataNode always READ_BLOCK or WRITE_BLOCK
> -----------------------------------------
>
>                 Key: HDFS-5720
>                 URL: https://issues.apache.org/jira/browse/HDFS-5720
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: datanode
>    Affects Versions: 2.2.0
>            Reporter: yangping wu
>            Priority: Trivial
>   Original Estimate: 480h
>  Remaining Estimate: 480h
>
> I find many following errors on my datanode's log. What is the reason for this and how
can I resolved it?
> {code}
> 2014-01-05 00:14:40,589 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing WRITE_BLOCK operation  src: /192.168.246.57:35455 dest: /192.168.246.75:50010
> java.io.IOException: Premature EOF from inputStream
>         at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
>         at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435)
>         at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-01-05 02:50:52,552 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing READ_BLOCK operation  src: /192.168.246.39:46693 dest: /192.168.246.75:50010
> org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for
BP-1398136447-192.168.246.60-1386067202761:blk_1089253308_1099605945003
>         at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:418)
>         at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:228)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:327)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:101)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-01-05 07:39:30,939 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode{data=FSDataset{dirpath='[/data1/hadoop/dfs/current,
/data2/hadoop/dfs/current, /data3/hadoop/dfs/current, /data4/hadoop/dfs/current, /data5/hadoop/dfs/current,
/data6/hadoop/dfs/current, /data7/hadoop/dfs/current, /data8/hadoop/dfs/current, /data9/hadoop/dfs/current,
/data10/hadoop/dfs/current, /data11/hadoop/dfs/current, /data12/hadoop/dfs/current]'}, localName='date51:50010',
storageID='DS-1312530819-192.168.246.75-50010-1388062043867', xmitsInProgress=0}:Exception
transfering block BP-1398136447-192.168.246.60-1386067202761:blk_1089401766_1099606093471
to mirror 192.168.242.51:50010: java.io.IOException: Connection reset by peer
> 2014-01-05 07:39:30,939 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
BP-1398136447-192.168.246.60-1386067202761:blk_1089401766_1099606093471 received exception
java.io.IOException: Connection reset by peer
> 2014-01-05 07:39:30,939 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing WRITE_BLOCK operation  src: /192.168.246.75:44779 dest: /192.168.246.75:50010
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>         at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1490)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:511)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-01-05 09:47:58,332 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing READ_BLOCK operation  src: /192.168.246.40:48386 dest: /192.168.246.75:50010
> java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be
ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.246.75:50010
remote=/192.168.246.40:48386]
>         at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
>         at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:172)
>         at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:220)
>         at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546)
>         at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:340)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:101)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
>         at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message