Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4CF3EE722 for ; Wed, 19 Dec 2012 16:39:11 +0000 (UTC) Received: (qmail 73434 invoked by uid 500); 19 Dec 2012 16:39:06 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 73283 invoked by uid 500); 19 Dec 2012 16:39:06 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 73275 invoked by uid 99); 19 Dec 2012 16:39:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 16:39:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert.rapplean@trueffect.com designates 213.199.154.207 as permitted sender) Received: from [213.199.154.207] (HELO am1outboundpool.messaging.microsoft.com) (213.199.154.207) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 16:38:58 +0000 Received: from mail78-am1-R.bigfish.com (10.3.201.248) by AM1EHSOBE023.bigfish.com (10.3.207.145) with Microsoft SMTP Server id 14.1.225.23; Wed, 19 Dec 2012 16:38:35 +0000 Received: from mail78-am1 (localhost [127.0.0.1]) by mail78-am1-R.bigfish.com (Postfix) with ESMTP id 2B9E24008F for ; Wed, 19 Dec 2012 16:38:35 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.245.85;KIP:(null);UIP:(null);IPV:NLI;H:CH1PRD0811HT001.namprd08.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -5 X-BigFish: PS-5(zz98dI9371I936eIc85fh542I4015Izz1de0h1202h1e76h1d1ah1d2ahzz177df4h17326ah8275dh18c673h1954cbh18602eh8275bhz2fh2a8h668h839hd25hf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh15d0h162dh1631h1758h1155h) Received-SPF: pass (mail78-am1: domain of trueffect.com designates 157.56.245.85 as permitted sender) client-ip=157.56.245.85; envelope-from=robert.rapplean@trueffect.com; helo=CH1PRD0811HT001.namprd08.prod.outlook.com ;.outlook.com ; Received: from mail78-am1 (localhost.localdomain [127.0.0.1]) by mail78-am1 (MessageSwitch) id 135593511269568_15747; Wed, 19 Dec 2012 16:38:32 +0000 (UTC) Received: from AM1EHSMHS006.bigfish.com (unknown [10.3.201.234]) by mail78-am1.bigfish.com (Postfix) with ESMTP id 033EE12006C for ; Wed, 19 Dec 2012 16:38:32 +0000 (UTC) Received: from CH1PRD0811HT001.namprd08.prod.outlook.com (157.56.245.85) by AM1EHSMHS006.bigfish.com (10.3.207.106) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 19 Dec 2012 16:38:24 +0000 Received: from CH1PRD0811MB430.namprd08.prod.outlook.com ([169.254.8.94]) by CH1PRD0811HT001.namprd08.prod.outlook.com ([10.255.155.36]) with mapi id 14.16.0245.002; Wed, 19 Dec 2012 16:38:24 +0000 From: Robert Rapplean To: "user@hadoop.apache.org" Subject: RE: hftp can list directories but won't send files Thread-Topic: hftp can list directories but won't send files Thread-Index: AQGals88zUSaRbXuXui8RT1ZTTksFQIi1tECAlI1XDkCQL+xrphRMmpA Date: Wed, 19 Dec 2012 16:38:23 +0000 Message-ID: <9240AD66F4BBBA4088ACAAC15099237B7E4C8C@CH1PRD0811MB430.namprd08.prod.outlook.com> References: <9240AD66F4BBBA4088ACAAC15099237B7E4565@CH1PRD0811MB430.namprd08.prod.outlook.com> <9240AD66F4BBBA4088ACAAC15099237B7E47AF@CH1PRD0811MB430.namprd08.prod.outlook.com> <7A606B54-8E90-47BF-A81D-1F7D9D85F9ED@hortonworks.com> In-Reply-To: <7A606B54-8E90-47BF-A81D-1F7D9D85F9ED@hortonworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [206.124.30.14] Content-Type: multipart/alternative; boundary="_000_9240AD66F4BBBA4088ACAAC15099237B7E4C8CCH1PRD0811MB430na_" MIME-Version: 1.0 X-OriginatorOrg: trueffect.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_9240AD66F4BBBA4088ACAAC15099237B7E4C8CCH1PRD0811MB430na_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks, Arpit. Didn't even know I was on Cloudera. I'll go bug them. Robert Rapplean Senior Software Engineer 303-872-2256 direct | 303.438.9597 main | www.trueffect.com From: Arpit Gupta [mailto:arpit@hortonworks.com] Sent: Tuesday, December 18, 2012 7:01 PM To: user@hadoop.apache.org Subject: Re: hftp can list directories but won't send files Robert Another thing you can try is export HADOOP_ROOT_LOGGER=3DDEBUG,console and run the hadoop dfs -cat comma= nd with hftp and you should get more logs on the client. Also since you are running cdh it might be better to ask on the cdh mailing= lists. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Dec 18, 2012, at 3:28 PM, Robert Rapplean > wrote: The cluster says this: Hadoop 2.0.0-cdh4.0.0 Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topd= ir/BUILD/hadoop-2.0.0-cdh4.0.0/src/hadoop-common-project/hadoop-common -r 5d678f6bb1f2b= c49e2287dd69ac41d7232fc9cdc Compiled by jenkins on Mon Jun 4 16:52:21 PDT 2012 >From source with checksum 64f877fc49f5adc0d7d55c13089e866e Which put a tiny strain on my knowledge to retrieve. Can you make a suggest= ion regarding which logs you want to look at? Robert Rapplean Senior Software Engineer 303-872-2256 direct | 303.438.9597 main | www.trueffect.com -----Original Message----- From: Harsh J [mailto:harsh@cloudera.com] Sent: Tuesday, December 18, 2012 4:17 PM To: > Subject: Re: hftp can list directories but won't send files What version/distribution of Hadoop is your source cluster? Also, I'd take a look at a your NN's and a few of your DN's logs right afte= r encountering this issue, to see the reason+stacktrace printed for the Ser= ver Error 500 (a code for a server-end fault). That'd give us more ideas on= whys. On Wed, Dec 19, 2012 at 4:13 AM, Robert Rapplean > wrote: Hey, everone. Just got finished reading about all of the unsubscribe messag= es in Sept-Oct, and was hoping someone has a clue about what my system is d= oing wrong. I suspect that this is a configuration issue, but I don't even = know where to start looking for it. I'm a developer, and my sysadmin is tie= d up until the end of the year. I'm trying to move files from one cluster to another using distcp, using th= e hftp protocol as specified in their instructions. I can read directories over hftp, but when I attempt to get a file I get a = 500 (internal server error). To eliminate the possibility of network and fi= rewall issues, I'm using hadoop fs -ls and hadoop fs -cat commands on the s= ource server in order to attempt to figure out this issue. This provides a directory of the files, which is correct. hadoop fs -ls ourlogs/day_id=3D19991231/hour_id=3D1999123123 -rw-r--r-- 3 username supergroup 812 2012-12-16 17:21 logfiles/day= _id=3D19991231/hour_id=3D1999123123/000008_0 This gives me a "file not found" error, which is also correct because the f= ile isn't there: hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D19991 231/hour_id=3D1999123123/000008_0x cat: `hftp://hdenn00.trueffect.com:50070/user/prodman/ods_fail/day_id=3D19991 231/hour_id=3D1999123123/000008_0x': No such file or directory This line gives me a 500 internal server error. The file is confirmed to be= on the server. hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D19991 231/hour_id=3D1999123123/000008_0 cat: HTTP_OK expected, received 500 Here is a stack trace of what distcp logs when I attempt this: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(Hf= tpFileSystem.java:365) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRange= InputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeI= nputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream= .java:187) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma= tion.java:1332) at org.apache.hadoop.mapred.Child.main(Child.java:262) Can someone tell me why hftp is failing to serve files, or at least where t= o look? -- Harsh J --_000_9240AD66F4BBBA4088ACAAC15099237B7E4C8CCH1PRD0811MB430na_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks, Arpit. Didn’= ;t even know I was on Cloudera. I’ll go bug them.

 <= /p>

Robert Rapplean

Senior Software Engineer<= o:p>

303-872-2256  direct=   | 303.438.9597  main | www.trueffect.com

 <= /p>

From: Arpit Gu= pta [mailto:arpit@hortonworks.com]
Sent: Tuesday, December 18, 2012 7:01 PM
To: user@hadoop.apache.org
Subject: Re: hftp can list directories but won't send files

 

Robert

 

Another thing you can try is

 

export HADOOP_ROOT_LOGGER=3DDEBUG,console and run th= e hadoop dfs -cat command with hftp and you should get more logs on the cli= ent. 

 

Also since you are running cdh it might be better to= ask on the cdh mailing lists.


--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/<= /span>

 

On Dec 18, 2012, at 3:28 PM, Robert Rapplean <robert.rapplean@trueffect.com= > wrote:



The cluster says this:

Hadoop 2.0.0-cdh4.0.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/ha= doop-2.0.0-cdh4.0.0/src/hadoop-common-project/hadoop-common -r 5d678f6b= b1f2bc49e2287dd69ac41d7232fc9cdc
Compiled by jenkins on Mon Jun  4 16:52:21 PDT 2012
>From source with checksum 64f877fc49f5adc0d7d55c13089e866e

Which put a tiny strain on my knowledge to retrieve. Can you make a suggest= ion regarding which logs you want to look at?

Robert Rapplean
Senior Software Engineer
303-872-2256  direct  | 303.438.9597  main | www.trueffect.com


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Tuesday, December 18, 2012 4:17 PM
To: <user@hadoop.apache.org>
Subject: Re: hftp can list directories but won't send files

What version/distribution of Hadoop is your source cluster?

Also, I'd take a look at a your NN's and a few of your DN's logs right afte= r encountering this issue, to see the reason+stacktrace printed for the= Server Error 500 (a code for a server-end fault). That'd give us more idea= s on whys.

On Wed, Dec 19, 2012 at 4:13 AM, Robert Rapplean <robert.rapplean@trueffect.com> wrote:

Hey, everone. Just go= t finished reading about all of the unsubscribe messages in Sept-Oct, and w= as hoping someone has a clue about what my system is doing wrong. I suspect= that this is a configuration issue, but I don't even know where to start looking for it. I'm a developer, and = my sysadmin is tied up until the end of the year.

I'm trying to move files from one cluster to another using distcp, using th= e hftp protocol as specified in their instructions.

I can read directories over hftp, but when I attempt to get a file I get a = 500 (internal server error). To eliminate the possibility of network and fi= rewall issues, I'm using hadoop fs -ls and hadoop fs -cat commands on the s= ource server in order to attempt to figure out this issue.

This provides a directory of the files, which is correct.

hadoop fs -ls ourlogs/day_id=3D19991231/hour_id=3D1999123123
-rw-r--r--   3 username supergroup      =   812 2012-12-16 17:21 logfiles/day_id=3D19991231/hour_id=3D19991= 23123/000008_0

This gives me a "file not found" error, which is also correct bec= ause the file isn't there:

hadoop fs -cat
hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id= =3D19991
231/hour_id=3D1999123123/000008_0x
cat:
`hftp://hdenn00.trueffect.com:50070/user/prodman/ods_fail/day_id=3D19991 231/hour_id=3D1999123123/000008_0x': No such file or directory

This line gives me a 500 internal server error. The file is confirmed to be= on the server.

hadoop fs -cat
hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D19991 231/hour_id=3D1999123123/000008_0
cat: HTTP_OK expected, received 500

Here is a stack trace of what distcp logs when I attempt this:

java.io.IOException: HTTP_OK expected, received 500
   at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOp= ener.connect(HftpFileSystem.java:365)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputS= tream(ByteRangeInputStream.java:119)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputSt= ream(ByteRangeInputStream.java:103)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteR= angeInputStream.java:187)
   at java.io.DataInputStream.read(DataInputStream.java:83)<= br>    at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(Di= stCp.java:424)
   at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(Dis= tCp.java:547)
   at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(Dis= tCp.java:314)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:= 50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.= java:393)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)=
   at org.apache.hadoop.mapred.Child$4.run(Child.java:268)    at java.security.AccessController.doPrivileged(Native Met= hod)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(U= serGroupInformation.java:1332)
   at org.apache.hadoop.mapred.Child.main(Child.java:262)
Can someone tell me why hftp is failing to serve files, or at least where t= o look?




--
Harsh J

 

--_000_9240AD66F4BBBA4088ACAAC15099237B7E4C8CCH1PRD0811MB430na_--