Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DD84417CA9 for ; Thu, 4 Jun 2015 15:40:48 +0000 (UTC) Received: (qmail 5249 invoked by uid 500); 4 Jun 2015 15:40:43 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 5143 invoked by uid 500); 4 Jun 2015 15:40:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 5133 invoked by uid 99); 4 Jun 2015 15:40:42 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Jun 2015 15:40:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 576571A44DD for ; Thu, 4 Jun 2015 15:40:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.991 X-Spam-Level: *** X-Spam-Status: No, score=3.991 tagged_above=-999 required=6.31 tests=[FREEMAIL_REPLY=1, HTML_MESSAGE=3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id piSQ83CUKD5y for ; Thu, 4 Jun 2015 15:40:35 +0000 (UTC) Received: from nk11p00mm-asmtp005.mac.com (nk11p00mm-asmtp005.mac.com [17.158.161.4]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id DB9F5275E7 for ; Thu, 4 Jun 2015 15:40:34 +0000 (UTC) Received: from MACProW7SSD (0-46.static.highlandsfibernetwork.com [216.9.0.46]) by nk11p00mm-asmtp005.mac.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Dec 4 2014)) with ESMTPSA id <0NPF00G44FJF6000@nk11p00mm-asmtp005.mac.com> for user@hadoop.apache.org; Thu, 04 Jun 2015 15:40:28 +0000 (GMT) From: Caesar Samsi To: user@hadoop.apache.org Subject: [SOLVED] RE: ack with firstBadLink as 192.168.1.12:50010? Date: Thu, 04 Jun 2015 11:40:20 -0400 Message-id: <090001d09edc$c408d010$4c1a7030$@mac.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="----=_NextPart_000_0901_01D09EBB.3CF8DDC0" X-Mailer: Microsoft Outlook 14.0 Thread-index: AdCe3FSC54nNSQIiQFedW5c3lxkkwg== Content-language: en-us X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151,1.0.33,0.0.0000 definitions=2015-06-04_08:2015-06-03,2015-06-04,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=1 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1412110000 definitions=main-1506040207 This is a multipart message in MIME format. ------=_NextPart_000_0901_01D09EBB.3CF8DDC0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable It was due to firewall configuration, port 50010 was not in the allow = list. =20 My cluster servers use ufw as the firewall, once I disabled it the error = went away and I was able to test run the cluster. =20 To disable ufw from the terminal: sudo ufw disable =20 Is there a list of ports needed by Hadoop to make firewall configuration = easier? =20 Many Thanks! Caesar. =20 From: Vishnu Viswanath [mailto:vishnu.viswanath25@gmail.com]=20 Sent: Thursday, June 04, 2015 1:51 AM To: user@hadoop.apache.org Subject: Re: ack with firstBadLink as 192.168.1.12:50010? =20 I had seen this issue. and it was due to data nodes not able to process = those many requests at a time.=20 =20 =20 On Thu, Jun 4, 2015 at 11:14 AM, Arpit Agarwal = wrote: I recall seeing this error due to a network misconfiguration. You may = want to verify that IP addresses and host names are correctly setup. =20 From: Caesar Samsi Reply-To: "user@hadoop.apache.org" Date: Wednesday, June 3, 2015 at 8:07 PM To: "user@hadoop.apache.org" Subject: ack with firstBadLink as 192.168.1.12:50010? =20 I=E2=80=99ve just built my distributed cluster but am getting the = following error when I try to use HDFS. =20 I=E2=80=99ve traced it by telnet to 192.168.1.12 50010 and it just waits = there waiting for a connection but never happens. =20 If I telnet on that host using localhost (127.0.0.1) the telnet = connection happens immediately. =20 What could be the cause? =20 >>=20 =20 hduser@hadoopmaster ~/hadoop $ hdfs dfs -copyFromLocal input input 15/06/03 20:03:36 INFO hdfs.DFSClient: Exception in = createBlockOutputStream java.io.IOException: Got error, status message , ack with firstBadLink = as 192.168.1.12:50010 at = org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkB= lockOpStatus(DataTransferProtoUtil.java:140) at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStre= am(DFSOutputStream.java:1334) at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream= (DFSOutputStream.java:1237) at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.j= ava:449) 15/06/03 20:03:36 INFO hdfs.DFSClient: Abandoning = BP-101149352-192.168.1.10-1433386347922:blk_1073741829_1005 15/06/03 20:03:36 INFO hdfs.DFSClient: Excluding datanode = DatanodeInfoWithStorage[192.168.1.12:50010,DS-1347a6fe-6bad-4df8-88cb-213= 78b847839,DISK] 15/06/03 20:03:36 WARN hdfs.DFSClient: Slow waitForAckedSeqno took = 70947ms (threshold=3D30000ms) =20 ------=_NextPart_000_0901_01D09EBB.3CF8DDC0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

It was due to firewall configuration, port 50010 was not in the allow = list.

 

My cluster servers use ufw as the firewall, once I disabled it the = error went away and I was able to test run the = cluster.

 

To disable ufw from the terminal: sudo ufw = disable

 

Is there a list of ports needed by Hadoop to make firewall = configuration easier?

 

Many Thanks! Caesar.

 

From:= = Vishnu Viswanath [mailto:vishnu.viswanath25@gmail.com]
Sent: = Thursday, June 04, 2015 1:51 AM
To: = user@hadoop.apache.org
Subject: Re: ack with firstBadLink as = 192.168.1.12:50010?

 

I had = seen this issue. and it was due to data nodes not able to process those = many requests at a time. 

 

 

On Thu, = Jun 4, 2015 at 11:14 AM, Arpit Agarwal <aagarwal@hortonworks.com> = wrote:

I recall seeing this error due to a network misconfiguration. You may = want to verify that IP addresses and host names are correctly = setup.

 

From: = Caesar = Samsi
Reply-To: "user@hadoop.apache.org"
Date: = Wednesday, June 3, 2015 at 8:07 PM
To: "user@hadoop.apache.org"
Subject: = ack with firstBadLink as 192.168.1.12:50010?

 

I=E2=80=99ve just built my distributed cluster but = am getting the following error when I try to use = HDFS.

 

I=E2=80=99ve traced it by telnet to 192.168.1.12 = 50010 and it just waits there waiting for a connection but never = happens.

 

If I telnet on that host using localhost = (127.0.0.1) the telnet connection happens = immediately.

 

What could be the cause?

 

>> 

 

hduser@hadoopmaster ~/hadoop $ hdfs dfs = -copyFromLocal input input

15/06/03 20:03:36 INFO hdfs.DFSClient: Exception = in createBlockOutputStream

java.io.IOException: Got error, status message , = ack with firstBadLink as 192.168.1.12:50010

        &nb= sp;       at = org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkB= lockOpStatus(DataTransferProtoUtil.java:140)

        &nb= sp;       at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStre= am(DFSOutputStream.java:1334)

        &nb= sp;       at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream= (DFSOutputStream.java:1237)

        &nb= sp;       at = org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.j= ava:449)

15/06/03 20:03:36 INFO hdfs.DFSClient: Abandoning = BP-101149352-192.168.1.10-1433386347922:blk_1073741829_1005

15/06/03 20:03:36 INFO hdfs.DFSClient: Excluding = datanode DatanodeInfoWithStorage[192.168.1.12:50010,DS-1347a6fe-6bad-4df8-88cb-21378= b847839,DISK]

15/06/03 20:03:36 WARN hdfs.DFSClient: Slow = waitForAckedSeqno took 70947ms = (threshold=3D30000ms)

=

 

------=_NextPart_000_0901_01D09EBB.3CF8DDC0--