Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 97E94D42D for ; Sun, 12 Aug 2012 02:06:58 +0000 (UTC) Received: (qmail 11757 invoked by uid 500); 12 Aug 2012 02:06:54 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 11653 invoked by uid 500); 12 Aug 2012 02:06:53 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 11645 invoked by uid 99); 12 Aug 2012 02:06:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Aug 2012 02:06:53 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Aug 2012 02:06:49 +0000 Received: by obbtb18 with SMTP id tb18so5828272obb.35 for ; Sat, 11 Aug 2012 19:06:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=dwgjgerF9bXDuPbzpWe5qr80Cg+SDWgRq1lC/SbdEFs=; b=Na2pvtC18t0nHP61b3u4B6KDku72FqPP8/nzZggwPw3jnNrDPoQriZmAPhLjq1wteV MG4/LHnGns6WiWnR/gWhEa1DMCgCAmzNOyPi0dt5thTQBox0IgZZsXNRvVAZCU+45iij 3xwLNElUlGMiCDdL8N9cDLOH9FiMAbpyihTicLwawOxdPjDUwLRw0dNsp66xzvc3XsOK DS2nn8j5ktTKV5KDeY11K2XMms5QhaMkFITrDp+h7RnB8mtFQl3D6YJWnmYM0+/9GUle wM4qQJ3na1ZU/OajG8hBz39aDFtuyf7yiyEZOWKuqJbmBIEWGYdJGgTPEnQFUiOstYJ0 qWBg== Received: by 10.60.29.169 with SMTP id l9mr6567779oeh.14.1344737188528; Sat, 11 Aug 2012 19:06:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.11.168 with HTTP; Sat, 11 Aug 2012 19:06:08 -0700 (PDT) In-Reply-To: References: From: Harsh J Date: Sun, 12 Aug 2012 07:36:08 +0530 Message-ID: Subject: Re: hftp in Hadoop 0.20.2 To: user@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQm/73tc73mgG3I0jOyQevOe1dfeNGQvV0CjHRwFHlxaIqJBAU/G/NbuQ9/FgGJoJmx3TY73 X-Virus-Checked: Checked by ClamAV on apache.org Jian, Do not rely on dfs.info.port, it is a deprecated property and does not exist anymore in 2.x releases. Rely instead on the fuller dfs.http.address in 1.x and dfs.namenode.http.address in 2.x. On Sat, Aug 11, 2012 at 3:45 AM, Jian Fang wrote: > Thanks Joey for the clarification. I will ask our hadoop admin to change > that. > But it would be great if this could be mentioned in the distcp document. > > Thanks, > > Jian > > > On Fri, Aug 10, 2012 at 6:06 PM, Joey Echeverria wrote: >> >> Yes, the dfs.info.port controls the HTTP port of the NN, including for >> HFTP. >> >> You should make sure that your settings for dfs.http.address and >> dfs.info.port are in sync. So change one of those to match the port >> number of the other. >> >> -Joey >> >> On Fri, Aug 10, 2012 at 5:41 PM, Jian Fang >> wrote: >> > Hi Joey, >> > >> > I run the following command and got the jetty port as 8023. >> > >> > $ grep "Jetty bound to port" >> > hadoop-hadoop-namenode-pnjhadoopnn01.barnesandnoble.com.log* >> > >> > hadoop-hadoop-namenode-pnjhadoopnn01.barnesandnoble.com.log.2012-04-07:2012-04-07 >> > 20:56:16,334 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port >> > 8023 >> > >> > Does this mean hftp is actually bound to port 8023? >> > >> > I am a bit confused. In hdfs-site.xml, we have the property defined as >> > follows. >> > >> > >> > >> > dfs.http.address >> > pnjhadoopnn01:50070 >> > >> > >> > and in core-site.xml, we have the following settings. >> > >> > >> > fs.default.name >> > pnjhadoopnn01:8020 >> > true >> > >> > >> > >> > dfs.secondary.info.port >> > 8022 >> > >> > >> > dfs.info.port >> > 8023 >> > >> > >> > mapred.job.tracker.info.port >> > 8024 >> > >> > >> > tasktracker.http.port >> > 8025 >> > >> > >> > mapred.job.tracker.info.port >> > 8024 >> > >> > >> > Does this mean hadoop honors dfs.info.port over dfs.http.address? >> > >> > Thanks, >> > >> > Jian >> > >> > On Fri, Aug 10, 2012 at 5:08 PM, Joey Echeverria >> > wrote: >> >> >> >> Can you post your NN logs? It looks like the NN is not actually >> >> started or is listening on another port for HTTP. >> >> >> >> -Joey >> >> >> >> On Fri, Aug 10, 2012 at 2:38 PM, Jian Fang >> >> wrote: >> >> > Already did that. Connection was rejected. >> >> > >> >> > >> >> > On Fri, Aug 10, 2012 at 2:24 PM, Joey Echeverria >> >> > wrote: >> >> >> >> >> >> Try: >> >> >> >> >> >> $ telnet pnjhadoopnn01 50070 >> >> >> >> >> >> -Joey >> >> >> >> >> >> On Fri, Aug 10, 2012 at 1:10 PM, Jian Fang >> >> >> wrote: >> >> >> > Here is the property in hdfs-site.xml >> >> >> > >> >> >> > >> >> >> > dfs.http.address >> >> >> > pnjhadoopnn01:50070 >> >> >> > >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Jian >> >> >> > >> >> >> > >> >> >> > On Fri, Aug 10, 2012 at 11:46 AM, Harsh J >> >> >> > wrote: >> >> >> >> >> >> >> >> Yes the test was to figure out if there really was a listener on >> >> >> >> 50070. Can you check the hdfs-site.xml on the NN machine for what >> >> >> >> its >> >> >> >> dfs.http.address may really be using for its port? >> >> >> >> >> >> >> >> On Fri, Aug 10, 2012 at 7:48 PM, Jian Fang >> >> >> >> wrote: >> >> >> >> > Hi Harsh, >> >> >> >> > >> >> >> >> > Seems the -p requires the root privilege, which I don't have. I >> >> >> >> > run >> >> >> >> > "netstat -a | grep 50070", but did not get back anything. As I >> >> >> >> > said, >> >> >> >> > telnet >> >> >> >> > did not work either. >> >> >> >> > >> >> >> >> > [hadoop@pnjhadoopnn01 ~]$ telnet pnjhadoopnn01 50070 >> >> >> >> > Trying xx.xx.xx.xx... >> >> >> >> > telnet: connect to address xx.xx.xx.xx: Connection refused >> >> >> >> > telnet: Unable to connect to remote host: Connection refused >> >> >> >> > >> >> >> >> > [hadoop@pnjhadoopnn01 ~]$ telnet localhost 50070 >> >> >> >> > Trying 127.0.0.1... >> >> >> >> > telnet: connect to address 127.0.0.1: Connection refused >> >> >> >> > telnet: Unable to connect to remote host: Connection refused >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > Jian >> >> >> >> > >> >> >> >> > On Fri, Aug 10, 2012 at 1:50 AM, Harsh J >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> Jian, >> >> >> >> >> >> >> >> >> >> From your NN, can you get us the output "netstat -anp | grep >> >> >> >> >> 50070"? >> >> >> >> >> >> >> >> >> >> On Fri, Aug 10, 2012 at 9:29 AM, Jian Fang >> >> >> >> >> wrote: >> >> >> >> >> > Thanks Harsh. But there is no firewall there, the two >> >> >> >> >> > clusters >> >> >> >> >> > are >> >> >> >> >> > on >> >> >> >> >> > the >> >> >> >> >> > same networks. I cannot telnet to the port even on the same >> >> >> >> >> > machine. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > On Thu, Aug 9, 2012 at 6:00 PM, Harsh J >> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> >> >> Hi Jian, >> >> >> >> >> >> >> >> >> >> >> >> HFTP is always-on by default. Can you check and make sure >> >> >> >> >> >> that >> >> >> >> >> >> the >> >> >> >> >> >> firewall isn't the cause of the connection refused on port >> >> >> >> >> >> 50070 >> >> >> >> >> >> on >> >> >> >> >> >> the NN and ports 50075 on the DNs here? >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Aug 10, 2012 at 1:47 AM, Jian Fang >> >> >> >> >> >> wrote: >> >> >> >> >> >> > Hi, >> >> >> >> >> >> > >> >> >> >> >> >> > We have a hadoop cluster of version 0.20.2 in production. >> >> >> >> >> >> > Now >> >> >> >> >> >> > we >> >> >> >> >> >> > have >> >> >> >> >> >> > another new Hadoop cluster using cloudera's CDH3U4. We >> >> >> >> >> >> > like >> >> >> >> >> >> > to >> >> >> >> >> >> > run >> >> >> >> >> >> > distcp to >> >> >> >> >> >> > copy files between the two clusters. Since the hadoop >> >> >> >> >> >> > versions >> >> >> >> >> >> > are >> >> >> >> >> >> > different, we have to use hftp protocol to copy files >> >> >> >> >> >> > based >> >> >> >> >> >> > on >> >> >> >> >> >> > the >> >> >> >> >> >> > hadoop >> >> >> >> >> >> > document here: >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > http://hadoop.apache.org/common/docs/r0.20.2/distcp.html#cpver. >> >> >> >> >> >> > >> >> >> >> >> >> > The problem is that I cannot access files via hftp from >> >> >> >> >> >> > the >> >> >> >> >> >> > current >> >> >> >> >> >> > production 0.20.2 cluster even though I can see the >> >> >> >> >> >> > following >> >> >> >> >> >> > setting >> >> >> >> >> >> > from >> >> >> >> >> >> > job tracker UI. >> >> >> >> >> >> > >> >> >> >> >> >> > dfs.http.address pnjhadoopnn01:50070 >> >> >> >> >> >> > >> >> >> >> >> >> > I tried to telnet this port, but got "connection refused" >> >> >> >> >> >> > error. >> >> >> >> >> >> > Seems >> >> >> >> >> >> > the >> >> >> >> >> >> > hftp service is not actually running. Could someone tell >> >> >> >> >> >> > me >> >> >> >> >> >> > how >> >> >> >> >> >> > to >> >> >> >> >> >> > enable >> >> >> >> >> >> > the hftp service in the 0.20.2 hadoop cluster so that I >> >> >> >> >> >> > can >> >> >> >> >> >> > run >> >> >> >> >> >> > distcp? >> >> >> >> >> >> > >> >> >> >> >> >> > Thanks in advance, >> >> >> >> >> >> > >> >> >> >> >> >> > John >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> Harsh J >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> Harsh J >> >> >> >> > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Harsh J >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Joey Echeverria >> >> >> Principal Solutions Architect >> >> >> Cloudera, Inc. >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Joey Echeverria >> >> Principal Solutions Architect >> >> Cloudera, Inc. >> > >> > >> >> >> >> -- >> Joey Echeverria >> Principal Solutions Architect >> Cloudera, Inc. > > -- Harsh J