hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sulabh choudhury <sula...@gmail.com>
Subject Re: DatanodeRegistration, socketTImeOutException
Date Fri, 11 May 2012 17:01:37 GMT
I have set dfs.datanode.max.xcievers=4096 and have swapping turned off,
Regionserver Heap = 24 GB
Datanode Heap = 1 GB
On Fri, May 11, 2012 at 9:55 AM, sulabh choudhury <sulabhc@gmail.com> wrote:

> I have spent a lot of time trying to find a solution to this issue, but
> have had no luck. I think this is because of the Hbase's read/write
> pattern, but I do not see any related errors in hbase logs.
> Does not look like it is because of a GC pause, but seeing several 480000
> ms timeOut certainly suggests something is really slowing down the *writes
> *( I do see this only in the write ch.)
>
> In my dataNode logs I see tonnes of
> 2012-05-11 09:34:30,953 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.10.2.102:50010,
> storageID=DS-1494937024-10.10.2.102-50010-1305755343443, infoPort=50075,
> ipcPort=50020):Got exception while serving
> blk_-5331817573170456741_12784653 to /10.10.2.102:
> java.net.SocketTimeoutException: 480000 millis timeout while waiting for
> channel to be ready for *write*. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.2.102:50010remote=/
> 10.10.2.102:46752]
>  at
> org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
> at
> org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
>  at
> org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:397)
>  at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:493)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:267)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:163)
>
> 2012-05-11 09:34:30,953 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.10.2.102:50010,
> storageID=DS-1494937024-10.10.2.102-50010-1305755343443, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.net.SocketTimeoutException: 480000 millis timeout while waiting for
> channel to be ready for write. ch :
> java.nio.channels.SocketChannel[connected local=/10.10.2.102:50010remote=/
> 10.10.2.102:46752]
>  at
> org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
> at
> org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
>  at
> org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:397)
>  at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:493)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:267)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:163)
>
>
> This block is mapped to a Hbase region, from NN logs :-
>
> 2012-05-10 15:46:35,117 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /hbase/table1/5a84f3844b7fd049c73a78b78ba6c2cf/.tmp/1639371300072460962.
> blk_4283960240517860151_12781124
> 2012-05-10 15:47:18,000 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.10.2.103:50010 is added
> to blk_4283960240517860151_12781124 size 134217728
> 2012-05-10 15:47:18,000 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.10.2.102:50010 is added
> to blk_4283960240517860151_12781124 size 134217728
>
>
>
> I am running hbase-0.90.4-cdh3u3 on hadoop-0.20.2-cdh3u3
>



-- 

-- 
Thanks and Regards,
Sulabh Choudhury

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message