hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amcn...@mcnabbs.org (Andrew McNabb)
Subject weird exceptions
Date Mon, 29 Jan 2007 18:48:35 GMT
Hi.  I'm getting some weird errors on the task trackers, which die as a
result.  I was wondering if this is a known problem.  These are showing
up in 0.10.0; I'm upgrading to 0.10.1 right now, but I thought I'd send
the logs along in case it's not something that's fixed in 0.10.1.  I
hope this is helpful.  Thank you.

07/01/29 08:11:00 INFO mapred.TaskTracker: Starting tracker tracker_m4a-3-24.local:50050
07/01/29 08:11:00 INFO ipc.Server: IPC Server handler 3 on 50050: starting
07/01/29 08:16:40 INFO mapred.TaskTracker: LaunchTaskAction: task_0001_m_000000_0
07/01/29 08:26:40 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:36:43 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:46:46 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:46:49 WARN fs.DFSClient: DFS Read: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Cannot open filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/.job.xml.crc
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:56:50 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:56:53 WARN fs.DFSClient: DFS Read: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Cannot open filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/.job.xml.crc
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:56:53 WARN fs.DataInputStream: Problem opening checksum file: /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/job.xml.
 Ignoring exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open
filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/.job.xml.crc
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:57:53 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:57:56 WARN fs.DFSClient: DFS Read: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Cannot open filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/job.xml
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:58:56 INFO fs.DFSClient: Could not obtain block from any node:  java.io.IOException:
No live nodes contain current block
07/01/29 08:58:59 WARN fs.DFSClient: DFS Read: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Cannot open filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/job.xml
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:58:59 WARN mapred.TaskTracker: Error initializing task_0001_m_000000_0:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open filename /tmp/hadoop.m17737/mapred/system/submit_k6c2r1/job.xml
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:210)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:258)

07/01/29 08:58:59 INFO mapred.TaskTracker: Recieved RenitTrackerAction from JobTracker
07/01/29 08:58:59 INFO mapred.TaskTracker: Error cleaning up task runner: java.lang.NullPointerException
        at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:1161)
        at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:1079)
        at org.apache.hadoop.mapred.TaskTracker.close(TaskTracker.java:397)
        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:841)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1468)

07/01/29 08:58:59 ERROR mapred.TaskTracker: Can not start task tracker because java.lang.NullPointerException
        at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:1087)
        at org.apache.hadoop.mapred.TaskTracker.close(TaskTracker.java:397)
        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:841)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1468)


-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868

Mime
View raw message