hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Falk <pe...@bugsoft.nu>
Subject Problem with socket timeouts
Date Thu, 15 Jul 2010 10:38:45 GMT
Hi,

We are noticing the following errors in our datanode logs. We are running
hbase on top of hdfs, but are not noticing any errors in the hbase logs. So
it seems like the hdfs clients are not suffering from these errors. However,
it would be nice to understand why they appear. We have upped the number of
xcievers to 1024 and are having some throughts about there are too many
sockets and some of them are timing out because they are not being used?
Also we have set dfs.datanode.socket.write.timeout and dfs.socket.timeout to
about 10 minutes. Anyone knows why these errors appears, and perhaps how to
get rid of them?

2010-07-15 12:31:02,307 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.10.54:50010,
storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
ipcPort=50020):Got exception while serving blk_3305049902326993023_98358 to
/192.168.10.54:
java.net.SocketTimeoutException: 600000 millis timeout while waiting for
channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010remote=/
192.168.10.54:46276]
        at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
        at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
        at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
        at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
        at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
        at java.lang.Thread.run(Thread.java:619)

2010-07-15 12:31:02,308 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.10.54:50010,
storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 600000 millis timeout while waiting for
channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010remote=/
192.168.10.54:46276]
        at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
        at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
        at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
        at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
        at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
        at java.lang.Thread.run(Thread.java:619)

TIA,
Peter

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message