hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chandravadana (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4026) bad coonect ack with first bad link
Date Mon, 29 Sep 2008 07:38:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635308#action_12635308
] 

chandravadana commented on HADOOP-4026:
---------------------------------------


sry, i don have the those logs right now..
i'll provide u with a new set with the same envt..

 bin/hadoop dfs -put /root/Desktop/srcdat input

08/09/29 12:42:58 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:42:58 INFO dfs.DFSClient: Abandoning block
blk_659235872296618299
08/09/29 12:43:05 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:43:05 INFO dfs.DFSClient: Abandoning block
blk_7875448137604385609

When i execute..

08/09/29 12:41:34 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:41:34 INFO dfs.DFSClient: Abandoning block
blk_4171702026185885604
08/09/29 12:41:40 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:41:40 INFO dfs.DFSClient: Abandoning block
blk_-7123364214947275818
08/09/29 12:41:46 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:41:46 INFO dfs.DFSClient: Abandoning block
blk_-2233035527027705705
08/09/29 12:41:52 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:41:52 INFO dfs.DFSClient: Abandoning block
blk_-6894859871092473815
08/09/29 12:41:58 INFO mapred.FileInputFormat: Total input paths to
process : 3
08/09/29 12:41:58 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:41:58 INFO dfs.DFSClient: Abandoning block
blk_8586133382152437359
08/09/29 12:42:22 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:42:22 INFO dfs.DFSClient: Abandoning block
blk_-8756555751747784250
08/09/29 12:42:28 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/09/29 12:42:28 INFO dfs.DFSClient: Abandoning block
blk_2494816095096186945
08/09/29 12:42:34 WARN dfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
        at org.apache.hadoop.dfs.DFSClient
$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2246)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700
(DFSClient.java:1702)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream
$DataStreamer.run(DFSClient.java:1842)

08/09/29 12:42:34 WARN dfs.DFSClient: Error Recovery for block
blk_2494816095096186945 bad datanode[1]
java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient
$DFSOutputStream.processDatanodeError(DFSClient.java:2081)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300
(DFSClient.java:1702)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream
$DataStreamer.run(DFSClient.java:1818)

Exception closing file /hadoop/hadoop-0.17.2.1/tmp/hadoop-
root/mapred/system/job_200809291237_0001/job.xml
java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient
$DFSOutputStream.processDatanodeError(DFSClient.java:2081)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300
(DFSClient.java:1702)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream
$DataStreamer.run(DFSClient.java:1818)


in 10.232.25.197 datanode's log

2008-09-29 12:41:34,349 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_4171702026185885604 src: /10.232.25.197:36669
dest: /10.232.25.197:50010
2008-09-29 12:41:34,351 INFO org.apache.hadoop.dfs.DataNode: writeBlock
blk_4171702026185885604 received exception
java.net.NoRouteToHostException: No route to host
2008-09-29 12:41:34,351 ERROR org.apache.hadoop.dfs.DataNode:
10.232.25.197:50010:DataXceiver: java.net.NoRouteToHostException: No
route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect
(SocketChannelImpl.java:574)
	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
	at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock
(DataNode.java:1150)
	at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:994)
	at java.lang.Thread.run(Thread.java:619)

2008-09-29 12:41:40,360 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-7123364214947275818 src: /10.232.25.197:36675
dest: /10.232.25.197:50010
2008-09-29 12:41:40,361 INFO org.apache.hadoop.dfs.DataNode: writeBlock
blk_-7123364214947275818 received exception
java.net.NoRouteToHostException: No route to host
2008-09-29 12:41:40,362 ERROR org.apache.hadoop.dfs.DataNode:
10.232.25.197:50010:DataXceiver: java.net.NoRouteToHostException: No
route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect
(SocketChannelImpl.java:574)
	at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
	at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock
(DataNode.java:1150)
	at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:994)
	at java.lang.Thread.run(Thread.java:619)

(this was present so many times..)



in 10.232.25.96's datanode log

2008-09-29 12:44:26,461 INFO org.apache.hadoop.dfs.DataNode:
10.232.25.96:50010 Starting thread to transfer block
blk_2148727769176979105 to 10.232.25.69:50010 2008-09-29 12:44:26,465
WARN org.apache.hadoop.dfs.DataNode: 10.232.25.96:50010:Failed to
transfer blk_2148727769176979105 to 10.232.25.69:50010 got
java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect
(SocketChannelImpl.java:574)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
        at org.apache.hadoop.dfs.DataNode$DataTransfer.run
(DataNode.java:2633)
        at java.lang.Thread.run(Thread.java:619)
2008-09-29 12:44:34,186 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_7136047761616343290 src: /10.232.25.197:36717
dest: /10.232.25.197:50010
2008-09-29 12:44:34,190 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_7136047761616343290 of size 396 from /10.232.25.197
2008-09-29 12:44:34,190 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_7136047761616343290 terminating
2008-09-29 12:44:41,466 INFO org.apache.hadoop.dfs.DataNode:
10.232.25.96:50010 Starting thread to transfer block
blk_2148727769176979105 to 10.232.25.69:50010
2008-09-29 12:44:41,467 WARN org.apache.hadoop.dfs.DataNode:
10.232.25.96:50010:Failed to transfer blk_2148727769176979105 to
10.232.25.69:50010 got java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect
(SocketChannelImpl.java:574)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
        at org.apache.hadoop.dfs.DataNode$DataTransfer.run
(DataNode.java:2633)
        at java.lang.Thread.run(Thread.java:619)


(....so many times)


in 10.232.25.69 datanode log

2008-09-29 12:32:27,341 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=DataNode, sessionId=null
2008-09-29 12:32:27,412 INFO org.apache.hadoop.dfs.DataNode: New storage
id DS-72368023-10.232.25.69-50010-1222671747405 is assigned to data-node
10.232.25.69:50010
2008-09-29 12:32:27,412 INFO org.apache.hadoop.dfs.DataNode:
10.232.25.69:50010In DataNode.run, data = FSDataset
{dirpath='/hadoop/hadoop-0.17.2.1/datanodedir/current'}
2008-09-29 12:32:27,415 INFO org.apache.hadoop.dfs.DataNode: using
BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 60000msec
2008-09-29 12:33:18,472 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of 0 blocks got processed in 27 msecs
2008-09-29 12:37:42,581 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of 0 blocks got processed in 2 msecs




This e-mail and any files transmitted with it are for the sole use of the intended recipient(s)
and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy
all copies of the original message. 
Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of
this email or any action taken in reliance on this e-mail is strictly 
prohibited and may be unlawful.


> bad coonect ack with first bad link
> -----------------------------------
>
>                 Key: HADOOP-4026
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4026
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.4
>         Environment: red hat Linux, cluster with 3 systems..
> 10.232.25.197- master
> 10.232.25.96-slave1
> 10.232.25.69-slave2
>            Reporter: chandravadana
>            Assignee: Hairong Kuang
>            Priority: Blocker
>   Original Estimate: 0.33h
>  Remaining Estimate: 0.33h
>
> wordcount/hi/ dir is the input dir 
> when i execute :
> # bin/hadoop dfs -copyFromLocal wordcount/hi wordcount/ins
> i get the foll msg
> 08/08/25 13:43:30 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:30 INFO dfs.DFSClient: Abandoning block
> blk_-3916191835981679734
> 08/08/25 13:43:36 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:36 INFO dfs.DFSClient: Abandoning block
> blk_-7058774921272589893
> 08/08/25 13:43:42 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:42 INFO dfs.DFSClient: Abandoning block
> blk_3767065959322874247
> 08/08/25 13:43:48 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:48 INFO dfs.DFSClient: Abandoning block
> blk_-8330992315825789947
> 08/08/25 13:43:54 WARN dfs.DFSClient: DataStreamer Exception:
> java.io.IOException: Unable to create new block.
> 08/08/25 13:43:54 WARN dfs.DFSClient: Error Recovery for block
> blk_-8330992315825789947 bad datanode[1]
> copyFromLocal: Could not get block locations. Aborting...
> when i examine the log file of the slave, i see this
> 2008-08-25 13:42:18,140 INFO org.apache.hadoop.dfs.DataNode:
> STARTUP_MSG: /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = slave1/10.232.25.96 STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.16.4 STARTUP_MSG:   build =
> http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
> 652614; compiled by 'hadoopqa' on Fri May  2 00:18:12 UTC 2008
> ************************************************************/
> 2008-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage: Storage
> directory /etc/hadoop_install/hadoop-0.16.4/datanodedir is not
> formatted.
> 2009-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage:
> Formatting ...
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Registered
> FSDatasetStatusMBean
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Opened
> server at 50010
> 2008-08-25 13:42:18,705 INFO org.apache.hadoop.dfs.DataNode: Balancing
> bandwith is 1048576 bytes/s
> 2008-08-25 13:42:18,911 INFO org.mortbay.util.Credential: Checking
> Resource aliases
> 2008-08-25 13:42:19,013 INFO org.mortbay.http.HttpServer: Version
> Jetty/5.1.4 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container:
> Started HttpContext[/static,/static]
> 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container: Started
> HttpContext[/logs,/logs]
> 2008-08-25 13:42:19,579 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.servlet.WebApplicationHandler@11ff436
> 2008-08-25 13:42:19,658 INFO org.mortbay.util.Container: Started
> WebApplicationContext[/,/]
> 2008-08-25 13:42:19,661 INFO org.mortbay.http.SocketListener: Started
> SocketListener on 0.0.0.0:50075
> 2008-08-25 13:42:19,661 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.Server@1b8f864
> 2008-08-25 13:42:19,706 INFO org.apache.hadoop.dfs.DataNode: New storage
> id DS-860242092-10.232.25.96-50010-1219651939700 is assigned to data-
> node 10.232.25.96:50010
> 2008-08-25 13:42:19,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=DataNode, sessionId=null
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode:
> 10.232.25.96:50010In DataNode.run, data = FSDataset
> {dirpath='/etc/hadoop_install/hadoop-0.16.4/datanodedir/current'}
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode: using
> BLOCKREPORT_INTERVAL of 3538776msec Initial delay: 60000msec
> 2008-08-25 13:42:19,828 INFO org.apache.hadoop.dfs.DataNode: BlockReport
> of 0 blocks got processed in 20 msecs
> 2008-08-25 13:45:43,982 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1031802361447574775 src: /10.232.25.197:40282
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,032 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1031802361447574775 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1031802361447574775 terminating
> 2008-08-25 13:45:44,105 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-1924738157193733587 src: /10.232.25.197:40285
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,106 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-1924738157193733587 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-1924738157193733587 terminating
> 2008-08-25 13:45:44,212 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_7001275375373078911 src: /10.232.25.197:40287
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,213 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_7001275375373078911 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_7001275375373078911 terminating
> 2008-08-25 13:45:44,277 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-7471693146363669981 src: /10.232.25.197:40289
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,278 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-7471693146363669981 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-7471693146363669981 terminating
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-6619078097753318750 src: /10.232.25.197:40291
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-6619078097753318750 of size 2778 from /10.232.25.197
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-6619078097753318750 terminating
> 2008-08-25 13:45:44,440 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1527614673854389960 src: /10.232.25.197:40293
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,441 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1527614673854389960 of size 4616 from /10.232.25.197
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1527614673854389960 terminating
> 2008-08-25 13:47:21,331 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_1527614673854389960
> 2008-08-25 13:48:11,458 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_7001275375373078911
> i don know what changes should i make n wer exactly the problem comes from.... 
> kindly help me in resolving this issue...
> Thanks in advance.. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message