hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C G <parallel...@yahoo.com>
Subject Re: Job hangs, strange error messages...help sought...
Date Tue, 30 Oct 2007 18:44:41 GMT
Each node runs 1 datanode instance.  I did notice this on "node4" trying to send to "node2":
  
2007-10-30 14:35:16,876 WARN org.apache.hadoop.dfs.DataNode: Failed to transfer blk_7852052048957161201
to 10.2.11.2:50010 got java.net.SocketExcep
tion: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.dfs.DataNode.sendBlock(DataNode.java:1231)
        at org.apache.hadoop.dfs.DataNode$DataTransfer.run(DataNode.java:1280)
        at java.lang.Thread.run(Thread.java:619)

  and then on node2:
  2007-10-30 14:35:16,918 INFO org.apache.hadoop.dfs.DataNode: Received block blk_7852052048957161201
from /10.2.11.3
2007-10-30 14:35:16,919 ERROR org.apache.hadoop.dfs.DataNode: DataXceiver: java.io.IOException:
Block blk_7852052048957161201 is valid, and cannot
be written to.
        at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
        at java.lang.Thread.run(Thread.java:619)

Ted Dunning <tdunning@veoh.com> wrote:
  
Is it possible you have some nodes running multiple instances of datanodes?


On 10/30/07 11:06 AM, "C G" 
wrote:

> Hi All:
> 
> Environment: 4 node grid running hadoop-0.14.1.
> 
> With the system shutdown I wiped out the old HDFS directory structure and
> created an empty directory. Did a namenode format, and then brought up the
> system with start-all.sh.
> 
> I then set up a directory structure, and ran the first job. The job runs
> 100% of the map jobs, completes ~ 87% of the reduce jobs, and then hangs.
> There are no user-level error messages. All systems go to idle.
> 
> I started looking at the Hadoop logs, first strange message from the
> namenode log:
> 
> 2007-10-30 13:48:01,991 WARN org.apache.hadoop.dfs.StateChange: DIR*
> NameSystem.startFile: failed to create file /import/raw_logs/20070929/_t
> ask_200710301345_0001_r_000001_0/part-00001 for
> DFSClient_task_200710301345_0001_r_000001_0 on client 10.2.11.4 because
> current leaseholder i
> s trying to recreate file.
> 2007-10-30 13:48:01,992 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 9 on 54310, call create(/import/raw_logs/20070929/_task_2007103
> 01345_0001_r_000001_0/part-00001, DFSClient_task_200710301345_0001_r_000001_0,
> true, 3, 67108864) from 10.2.11.4:34016: error: org.apache.had
> oop.dfs.AlreadyBeingCreatedException: failed to create file
> /import/raw_logs/20070929/_task_200710301345_0001_r_000001_0/part-00001 for
> DFSCl
> ient_task_200710301345_0001_r_000001_0 on client 10.2.11.4 because current
> leaseholder is trying to recreate file.
> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create file
> /import/raw_logs/20070929/_task_200710301345_0001_r_000001_0/part-0
> 0001 for DFSClient_task_200710301345_0001_r_000001_0 on client 10.2.11.4
> because current leaseholder is trying to recreate file.
> at 
> org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:788)
> at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:725)
> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:307)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> 
> Second strange error message comes from the jobtracker log:
> 2007-10-30 13:59:26,190 INFO org.apache.hadoop.mapred.JobTracker: Ignoring
> 'duplicate' heartbeat from 'tracker_localhost.localdomain:50050'
> 
> I'm curious about how to proceed. I suspect that my code is OK as I've run
> it numerous times in both single node and multiple grid environments. I've
> never seen these error messages before.
> 
> Any help much appreciated....
> 
> Thanks,
> C G 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com 



 __________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message