hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dru Jensen <drujen...@gmail.com>
Subject xceiverCount 257 exceeds the limit of concurrent xcievers 256
Date Wed, 12 Nov 2008 21:19:31 GMT
hbase-users,

I have been running MR processes for several days against HBase with  
success until recently the region servers shut themselves down.
Hadoop 0.18.1
Hbase 0.18.1
3 node cluster

Checking the region server logs, I see the following Exception before  
shutdown:

2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:  
Exception in createBlockOutputStream java.io.IOException: Could not  
read from stream
2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:  
Abandoning block blk_-5467014108758633036_595771
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient:  
DataStreamer Exception: java.io.IOException: Unable to create new block.
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient: Error  
Recovery for block blk_-5467014108758633036_595771 bad datanode[0]
2008-11-11 19:55:58,482 FATAL  
org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required.  
Forcing server shutdown

Hadoop datanode:

ERROR org.apache.hadoop.dfs.DataNode:  
DatanodeRegistration(10.10.10.53:50010,  
storageID=DS-1570581820-10.10.10.53-50010-1224117842339,  
infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:  
xceiverCount 258 exceeds the limit of concurrent xcievers 256

Also, I can't seem to recover from this.

At restart HBase regionserver log:

ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error  
opening region table,key,1226427697843
java.io.IOException: Could not obtain block:  
blk_-7717647125888341834_589063 file=/hbase/table/1691377931/title/ 
mapfiles/7939912624709782499/index

Hadoop datanode:

ERROR org.apache.hadoop.dfs.DataNode:  
DatanodeRegistration(10.10.10.51:50010,  
storageID=DS-2079833884-10.10.10.51-50010-1224117832456,  
infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:  
xceiverCount 257 exceeds the limit of concurrent xcievers 256
	at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1030)
	at java.lang.Thread.run(Thread.java:595)

Any help is appreciated.

Thanks,
Dru


Mime
View raw message