hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Ruchovets <oruchov...@gmail.com>
Subject Re: java.io.IOException: Bad connect ack with firstBadLink
Date Tue, 09 Nov 2010 11:07:41 GMT
What does it mean:   It looks like second attempt to process data after the
first time it failed?
All Task AttemptsTask AttemptsMachineStatusProgressStart TimeShuffle
FinishedSort FinishedFinish TimeErrorsTask LogsCountersActions
attempt_201011081008_0002_r_000008_0/default-rack/<http://hadoop1.infolinks.local:8022>
FAILED0.00%9-Nov-2010 04:36:159-Nov-2010 05:09:08 (32mins, 52sec)9-Nov-2010
05:09:08 (0sec)9-Nov-2010 05:14:07 (37mins, 51sec)

java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)

Last 4KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-4097>
Last 8KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-8193>
All<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&all=true>
10</taskstats.jsp?jobid=job_201011081008_0002&tipid=task_201011081008_0002_r_000008&taskid=attempt_201011081008_0002_r_000008_0>



attempt_201011081008_0002_r_000008_1/default-rack/<http://hadoop-transfer.infolinks.local:8022>
RUNNING24.98%9-Nov-2010 05:50:21




On Tue, Nov 9, 2010 at 12:58 PM, Oleg Ruchovets <oruchovets@gmail.com>wrote:

> Hi , running hadoop map/reduce got such exception?
>
>   1) Why does it happen?
>
>   2) Job didn't failed and continue it's execution? Does this exception cause losing
data or map/reduce
>
> uses recovery mechanism?
>
>
>
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-9208177033562590356_775948
> 2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target
node: 10.11.87.61:50010
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready
for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.87.61:55309 remote=/10.11.87.61:50010]
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-10251707095594311_775950
> 2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target
node: 10.11.87.61:50010
> 2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3509928762116143133_775950
> 2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target
node: 10.11.87.61:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3138002906377068146_775950
> 2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target
node: 10.11.87.61:50010
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812)
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block
blk_3138002906377068146_775950 bad datanode[2] nodes == null
> 2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations.
Source file "/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_000008_0/part-r-00008"
- Aborting...
> 2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
> 2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for
the task
>
>
> Thanks
>
> Oleg.
>
> ------------------------------
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message