hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4026) bad coonect ack with first bad link
Date Mon, 29 Sep 2008 23:42:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635599#action_12635599
] 

Hairong Kuang commented on HADOOP-4026:
---------------------------------------

I saw a lot of java.net.NoRouteToHostExceptions ere thrown at some datanodes. This kind of
exception normally means that an error occurred while attempting to connect a socket to a
remote address and port. Typically, the remote host cannot be reached because of an intervening
firewall, or if an intermediate router is down.  Could you please check, for example, if you
could connect to 10.232.25.69 at port 50010 from 10.232.25.96?



> bad coonect ack with first bad link
> -----------------------------------
>
>                 Key: HADOOP-4026
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4026
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.4
>         Environment: red hat Linux, cluster with 3 systems..
> 10.232.25.197- master
> 10.232.25.96-slave1
> 10.232.25.69-slave2
>            Reporter: chandravadana
>            Assignee: Hairong Kuang
>            Priority: Blocker
>   Original Estimate: 0.33h
>  Remaining Estimate: 0.33h
>
> wordcount/hi/ dir is the input dir 
> when i execute :
> # bin/hadoop dfs -copyFromLocal wordcount/hi wordcount/ins
> i get the foll msg
> 08/08/25 13:43:30 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:30 INFO dfs.DFSClient: Abandoning block
> blk_-3916191835981679734
> 08/08/25 13:43:36 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:36 INFO dfs.DFSClient: Abandoning block
> blk_-7058774921272589893
> 08/08/25 13:43:42 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:42 INFO dfs.DFSClient: Abandoning block
> blk_3767065959322874247
> 08/08/25 13:43:48 INFO dfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.232.25.69:50010
> 08/08/25 13:43:48 INFO dfs.DFSClient: Abandoning block
> blk_-8330992315825789947
> 08/08/25 13:43:54 WARN dfs.DFSClient: DataStreamer Exception:
> java.io.IOException: Unable to create new block.
> 08/08/25 13:43:54 WARN dfs.DFSClient: Error Recovery for block
> blk_-8330992315825789947 bad datanode[1]
> copyFromLocal: Could not get block locations. Aborting...
> when i examine the log file of the slave, i see this
> 2008-08-25 13:42:18,140 INFO org.apache.hadoop.dfs.DataNode:
> STARTUP_MSG: /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = slave1/10.232.25.96 STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.16.4 STARTUP_MSG:   build =
> http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
> 652614; compiled by 'hadoopqa' on Fri May  2 00:18:12 UTC 2008
> ************************************************************/
> 2008-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage: Storage
> directory /etc/hadoop_install/hadoop-0.16.4/datanodedir is not
> formatted.
> 2009-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage:
> Formatting ...
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Registered
> FSDatasetStatusMBean
> 2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Opened
> server at 50010
> 2008-08-25 13:42:18,705 INFO org.apache.hadoop.dfs.DataNode: Balancing
> bandwith is 1048576 bytes/s
> 2008-08-25 13:42:18,911 INFO org.mortbay.util.Credential: Checking
> Resource aliases
> 2008-08-25 13:42:19,013 INFO org.mortbay.http.HttpServer: Version
> Jetty/5.1.4 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container:
> Started HttpContext[/static,/static]
> 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container: Started
> HttpContext[/logs,/logs]
> 2008-08-25 13:42:19,579 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.servlet.WebApplicationHandler@11ff436
> 2008-08-25 13:42:19,658 INFO org.mortbay.util.Container: Started
> WebApplicationContext[/,/]
> 2008-08-25 13:42:19,661 INFO org.mortbay.http.SocketListener: Started
> SocketListener on 0.0.0.0:50075
> 2008-08-25 13:42:19,661 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.Server@1b8f864
> 2008-08-25 13:42:19,706 INFO org.apache.hadoop.dfs.DataNode: New storage
> id DS-860242092-10.232.25.96-50010-1219651939700 is assigned to data-
> node 10.232.25.96:50010
> 2008-08-25 13:42:19,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=DataNode, sessionId=null
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode:
> 10.232.25.96:50010In DataNode.run, data = FSDataset
> {dirpath='/etc/hadoop_install/hadoop-0.16.4/datanodedir/current'}
> 2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode: using
> BLOCKREPORT_INTERVAL of 3538776msec Initial delay: 60000msec
> 2008-08-25 13:42:19,828 INFO org.apache.hadoop.dfs.DataNode: BlockReport
> of 0 blocks got processed in 20 msecs
> 2008-08-25 13:45:43,982 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1031802361447574775 src: /10.232.25.197:40282
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,032 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1031802361447574775 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1031802361447574775 terminating
> 2008-08-25 13:45:44,105 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-1924738157193733587 src: /10.232.25.197:40285
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,106 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-1924738157193733587 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-1924738157193733587 terminating
> 2008-08-25 13:45:44,212 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_7001275375373078911 src: /10.232.25.197:40287
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,213 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_7001275375373078911 of size 3161 from /10.232.25.197
> 2008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_7001275375373078911 terminating
> 2008-08-25 13:45:44,277 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-7471693146363669981 src: /10.232.25.197:40289
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,278 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-7471693146363669981 of size 6628 from /10.232.25.197
> 2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-7471693146363669981 terminating
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_-6619078097753318750 src: /10.232.25.197:40291
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_-6619078097753318750 of size 2778 from /10.232.25.197
> 2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_-6619078097753318750 terminating
> 2008-08-25 13:45:44,440 INFO org.apache.hadoop.dfs.DataNode: Receiving
> block blk_1527614673854389960 src: /10.232.25.197:40293
> dest: /10.232.25.197:50010
> 2008-08-25 13:45:44,441 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
> forwarding connect ack to upstream firstbadlink is
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode: Received
> block blk_1527614673854389960 of size 4616 from /10.232.25.197
> 2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_1527614673854389960 terminating
> 2008-08-25 13:47:21,331 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_1527614673854389960
> 2008-08-25 13:48:11,458 INFO org.apache.hadoop.dfs.DataBlockScanner:
> Verification succeeded for blk_7001275375373078911
> i don know what changes should i make n wer exactly the problem comes from.... 
> kindly help me in resolving this issue...
> Thanks in advance.. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message