hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Marz <nat...@rapleaf.com>
Subject LeaseExpiredException and too many xceiver
Date Fri, 31 Oct 2008 21:49:49 GMT
Hello,

We are seeing some really bad errors on our hadoop cluster. After  
reformatting the whole cluster, the first job we run immediately fails  
with "Could not find block locations..." errrors. In the namenode  
logs, we see a ton of errors like:

2008-10-31 14:20:44,799 INFO org.apache.hadoop.ipc.Server: IPC Server  
handler 5 on 7276, call addBlock(/tmp/dustintmp/shredded_dataunits/_t$
org.apache.hadoop.dfs.LeaseExpiredException: No lease on /tmp/ 
dustintmp/shredded_dataunits/_temporary/ 
_attempt_200810311418_0002_m_000023_0$
         at  
org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1166)
         at  
org 
.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java: 
1097)
         at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
         at  
sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 
25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)



In the datanode logs, we see a ton of errors like:

2008-10-31 14:20:09,978 ERROR org.apache.hadoop.dfs.DataNode:  
DatanodeRegistration(10.100.11.115:50010,  
storageID=DS-2129547091-10.100.11.1$
of concurrent xcievers 256
         at org.apache.hadoop.dfs.DataNode 
$DataXceiver.run(DataNode.java:1030)
         at java.lang.Thread.run(Thread.java:619)



Anyone have any ideas on what may be wrong?

Thanks,
Nathan Marz
Rapleaf

Mime
View raw message