hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From fred wang <fredwang...@gmail.com>
Subject <sorry if duplicate> help needed: mapred Taskrunner(hadoop 0.20.1) was marked as successful but with commuication error at random
Date Wed, 02 Dec 2009 13:10:24 GMT
 It occures in our hadoop cluster)(0.20.1 version) from time to time(the log
is below). Anyone could help? what hadoop is doing in the course of(after)
committing before the map task is done? joWhat kind of reasons could make
communication error. The job is still regarded as succeed even with this
communication error. Thanks for the help.


2009-11-25 01:21:30,476 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:30,541 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2009-11-25 01:21:30,559 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
...
2009-11-25 01:21:31,616 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 27
2009-11-25 01:21:31,624 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:31,627 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: data buffer =
79691776/99614720
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: record buffer
= 262144/327680
2009-11-25 01:21:33,266 INFO org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2009-11-25 01:21:33,277 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,187 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 0
2009-11-25 01:21:37,236 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,271 INFO org.apache.hadoop.mapred.TaskRunner:
Task:attempt_200911171829_0181_m_000008_0 is done. And is in the process of
commiting
2009-11-25 01:22:37,323 INFO org.apache.hadoop.mapred.TaskRunner:
Communication exception: java.io.IOException: Call to
/127.0.0.1:53674failed on local exception:
java.nio.channels.ClosedByInterruptException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy0.statusUpdate(Unknown Source)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:543)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
... 4 more
2009-11-25 01:22:37,338 INFO org.apache.hadoop.mapred.TaskRunner: Task
'attempt_200911171829_0181_m_000008_0' done.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message