hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chandravadana <chandravadana.selvach...@cognizant.com>
Subject bad connection ack with firstbadlink 10.23.....
Date Mon, 25 Aug 2008 08:48:03 GMT

hi..

i'm using a cluster of 3 systems.. 1 being the master and the other 2
are slaves.

i don get any errors when i cluster 2 systems. when i add the 3rd one, i
get the following error..

master: 10.232.25.197
slave1: 10.232.25.96
slave2:10.232.25.69

wordcount/hi/ dir is the input dir

when i execute :

# bin/hadoop dfs -copyFromLocal wordcount/hi wordcount/ins

08/08/25 13:43:30 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/08/25 13:43:30 INFO dfs.DFSClient: Abandoning block
blk_-3916191835981679734
08/08/25 13:43:36 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/08/25 13:43:36 INFO dfs.DFSClient: Abandoning block
blk_-7058774921272589893
08/08/25 13:43:42 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/08/25 13:43:42 INFO dfs.DFSClient: Abandoning block
blk_3767065959322874247
08/08/25 13:43:48 INFO dfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.232.25.69:50010
08/08/25 13:43:48 INFO dfs.DFSClient: Abandoning block
blk_-8330992315825789947
08/08/25 13:43:54 WARN dfs.DFSClient: DataStreamer Exception:
java.io.IOException: Unable to create new block.
08/08/25 13:43:54 WARN dfs.DFSClient: Error Recovery for block
blk_-8330992315825789947 bad datanode[1]
copyFromLocal: Could not get block locations. Aborting...

when i examine the log file of the slave, i see this

2008-08-25 13:42:18,140 INFO org.apache.hadoop.dfs.DataNode:
STARTUP_MSG: /************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = slave1/10.232.25.96 STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.16.4 STARTUP_MSG:   build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r
652614; compiled by 'hadoopqa' on Fri May  2 00:18:12 UTC 2008
************************************************************/
2008-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage: Storage
directory /etc/hadoop_install/hadoop-0.16.4/datanodedir is not
formatted.
2009-08-25 13:42:18,634 INFO org.apache.hadoop.dfs.Storage:
Formatting ...
2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Registered
FSDatasetStatusMBean
2008-08-25 13:42:18,701 INFO org.apache.hadoop.dfs.DataNode: Opened
server at 50010
2008-08-25 13:42:18,705 INFO org.apache.hadoop.dfs.DataNode: Balancing
bandwith is 1048576 bytes/s
2008-08-25 13:42:18,911 INFO org.mortbay.util.Credential: Checking
Resource aliases
2008-08-25 13:42:19,013 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4 2008-08-25 13:42:19,014 INFO org.mortbay.util.Container:
Started HttpContext[/static,/static]
2008-08-25 13:42:19,014 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2008-08-25 13:42:19,579 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@11ff436
2008-08-25 13:42:19,658 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2008-08-25 13:42:19,661 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:50075
2008-08-25 13:42:19,661 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.Server@1b8f864
2008-08-25 13:42:19,706 INFO org.apache.hadoop.dfs.DataNode: New storage
id DS-860242092-10.232.25.96-50010-1219651939700 is assigned to data-
node 10.232.25.96:50010
2008-08-25 13:42:19,733 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=DataNode, sessionId=null
2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode:
10.232.25.96:50010In DataNode.run, data = FSDataset
{dirpath='/etc/hadoop_install/hadoop-0.16.4/datanodedir/current'}
2008-08-25 13:42:19,755 INFO org.apache.hadoop.dfs.DataNode: using
BLOCKREPORT_INTERVAL of 3538776msec Initial delay: 60000msec
2008-08-25 13:42:19,828 INFO org.apache.hadoop.dfs.DataNode: BlockReport
of 0 blocks got processed in 20 msecs
2008-08-25 13:45:43,982 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_1031802361447574775 src: /10.232.25.197:40282
dest: /10.232.25.197:50010
2008-08-25 13:45:44,032 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_1031802361447574775 of size 3161 from /10.232.25.197
2008-08-25 13:45:44,081 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_1031802361447574775 terminating
2008-08-25 13:45:44,105 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-1924738157193733587 src: /10.232.25.197:40285
dest: /10.232.25.197:50010
2008-08-25 13:45:44,106 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_-1924738157193733587 of size 6628 from /10.232.25.197
2008-08-25 13:45:44,193 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_-1924738157193733587 terminating
2008-08-25 13:45:44,212 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_7001275375373078911 src: /10.232.25.197:40287
dest: /10.232.25.197:50010
2008-08-25 13:45:44,213 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_7001275375373078911 of size 3161 from /10.232.25.197
2008-08-25 13:45:44,256 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_7001275375373078911 terminating
2008-08-25 13:45:44,277 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-7471693146363669981 src: /10.232.25.197:40289
dest: /10.232.25.197:50010
2008-08-25 13:45:44,278 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_-7471693146363669981 of size 6628 from /10.232.25.197
2008-08-25 13:45:44,362 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_-7471693146363669981 terminating
2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-6619078097753318750 src: /10.232.25.197:40291
dest: /10.232.25.197:50010
2008-08-25 13:45:44,380 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_-6619078097753318750 of size 2778 from /10.232.25.197
2008-08-25 13:45:44,424 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_-6619078097753318750 terminating
2008-08-25 13:45:44,440 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_1527614673854389960 src: /10.232.25.197:40293
dest: /10.232.25.197:50010
2008-08-25 13:45:44,441 INFO org.apache.hadoop.dfs.DataNode: Datanode 0
forwarding connect ack to upstream firstbadlink is
2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode: Received
block blk_1527614673854389960 of size 4616 from /10.232.25.197
2008-08-25 13:45:44,526 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_1527614673854389960 terminating
2008-08-25 13:47:21,331 INFO org.apache.hadoop.dfs.DataBlockScanner:
Verification succeeded for blk_1527614673854389960
2008-08-25 13:48:11,458 INFO org.apache.hadoop.dfs.DataBlockScanner:
Verification succeeded for blk_7001275375373078911


i should i do.. kindly help me in resolving this issue...
Thanks in advance..



This e-mail and any files transmitted with it are for the sole use of the intended recipient(s)
and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy
all copies of the original message. 
Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of
this email or any action taken in reliance on this e-mail is strictly 
prohibited and may be unlawful.

Mime
View raw message