Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 84E42D82A for ; Tue, 18 Dec 2012 23:06:34 +0000 (UTC) Received: (qmail 56862 invoked by uid 500); 18 Dec 2012 23:06:29 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 56686 invoked by uid 500); 18 Dec 2012 23:06:29 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56677 invoked by uid 99); 18 Dec 2012 23:06:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 23:06:29 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert.rapplean@trueffect.com designates 216.32.180.16 as permitted sender) Received: from [216.32.180.16] (HELO va3outboundpool.messaging.microsoft.com) (216.32.180.16) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 23:06:21 +0000 Received: from mail248-va3-R.bigfish.com (10.7.14.249) by VA3EHSOBE011.bigfish.com (10.7.40.61) with Microsoft SMTP Server id 14.1.225.23; Tue, 18 Dec 2012 23:05:58 +0000 Received: from mail248-va3 (localhost [127.0.0.1]) by mail248-va3-R.bigfish.com (Postfix) with ESMTP id 04667700080 for ; Tue, 18 Dec 2012 23:05:58 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.245.85;KIP:(null);UIP:(null);IPV:NLI;H:CH1PRD0811HT001.namprd08.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -4 X-BigFish: PS-4(zz98dI9371I936eIc85fh1447Izz1de0h1202h1e76h1d1ah1d2ahzz177df4h18c673h17326ah8275bh8275dhz2fh2a8h668h839hd25hf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh15d0h162dh1631h1758h1155h) Received-SPF: pass (mail248-va3: domain of trueffect.com designates 157.56.245.85 as permitted sender) client-ip=157.56.245.85; envelope-from=robert.rapplean@trueffect.com; helo=CH1PRD0811HT001.namprd08.prod.outlook.com ;.outlook.com ; Received: from mail248-va3 (localhost.localdomain [127.0.0.1]) by mail248-va3 (MessageSwitch) id 135587195523248_19846; Tue, 18 Dec 2012 23:05:55 +0000 (UTC) Received: from VA3EHSMHS022.bigfish.com (unknown [10.7.14.247]) by mail248-va3.bigfish.com (Postfix) with ESMTP id 03CC3C4004A for ; Tue, 18 Dec 2012 23:05:55 +0000 (UTC) Received: from CH1PRD0811HT001.namprd08.prod.outlook.com (157.56.245.85) by VA3EHSMHS022.bigfish.com (10.7.99.32) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 18 Dec 2012 23:05:51 +0000 Received: from CH1PRD0811MB430.namprd08.prod.outlook.com ([169.254.8.94]) by CH1PRD0811HT001.namprd08.prod.outlook.com ([10.255.155.36]) with mapi id 14.16.0245.002; Tue, 18 Dec 2012 23:05:51 +0000 From: Robert Rapplean To: "user@hadoop.apache.org" Subject: RE: hftp can list directories but won't send files Thread-Topic: hftp can list directories but won't send files Thread-Index: AQGals88zUSaRbXuXui8RT1ZTTksFQF5raccmHnrp0A= Date: Tue, 18 Dec 2012 23:05:50 +0000 Message-ID: <9240AD66F4BBBA4088ACAAC15099237B7E4791@CH1PRD0811MB430.namprd08.prod.outlook.com> References: <9240AD66F4BBBA4088ACAAC15099237B7E4565@CH1PRD0811MB430.namprd08.prod.outlook.com> <89F44B63-CE39-4D0C-A86B-D7E26641D672@hortonworks.com> In-Reply-To: <89F44B63-CE39-4D0C-A86B-D7E26641D672@hortonworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-antivirus: AVG for E-mail x-avg-id: ID323FE9A1-2899855F x-originating-ip: [206.124.30.14] Content-Type: multipart/alternative; boundary="_000_9240AD66F4BBBA4088ACAAC15099237B7E4791CH1PRD0811MB430na_" MIME-Version: 1.0 X-OriginatorOrg: trueffect.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_9240AD66F4BBBA4088ACAAC15099237B7E4791CH1PRD0811MB430na_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks for the reply, Arpit. Yes, both of those produce a correct response, although the second one's sy= ntax is: hadoop fs -cat logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0 From: Arpit Gupta [mailto:arpit@hortonworks.com] Sent: Tuesday, December 18, 2012 3:49 PM Hi Robert Does the cat work for you if you dont use hftp, something like hadoop fs -cat hdfs://hdenn00.trueffect.com:8020/user/username/logfiles/day= _id=3D19991231/hour_id=3D1999123123/000008_0x Or hadoop fs -cat /user/username/logfiles/day_id=3D19991231/hour_id=3D19991231= 23/000008_0x -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Dec 18, 2012, at 2:43 PM, Robert Rapplean > wrote: Hey, everone. Just got finished reading about all of the unsubscribe messag= es in Sept-Oct, and was hoping someone has a clue about what my system is d= oing wrong. I suspect that this is a configuration issue, but I don't even = know where to start looking for it. I'm a developer, and my sysadmin is tie= d up until the end of the year. I'm trying to move files from one cluster to another using distcp, using th= e hftp protocol as specified in their instructions. I can read directories over hftp, but when I attempt to get a file I get a = 500 (internal server error). To eliminate the possibility of network and fi= rewall issues, I'm using hadoop fs -ls and hadoop fs -cat commands on the s= ource server in order to attempt to figure out this issue. This provides a directory of the files, which is correct. hadoop fs -ls ourlogs/day_id=3D19991231/hour_id=3D1999123123 -rw-r--r-- 3 username supergroup 812 2012-12-16 17:21 logfiles/day= _id=3D19991231/hour_id=3D1999123123/000008_0 This gives me a "file not found" error, which is also correct because the f= ile isn't there: hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/da= y_id=3D19991231/hour_id=3D1999123123/000008_0x cat: `hftp://hdenn00.trueffect.com:50070/user/prodman/ods_fail/day_id=3D199= 91231/hour_id=3D1999123123/000008_0x': No such file or directory This line gives me a 500 internal server error. The file is confirmed to be= on the server. hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/da= y_id=3D19991231/hour_id=3D1999123123/000008_0 cat: HTTP_OK expected, received 500 Here is a stack trace of what distcp logs when I attempt this: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(Hf= tpFileSystem.java:365) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRange= InputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeI= nputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream= .java:187) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma= tion.java:1332) at org.apache.hadoop.mapred.Child.main(Child.java:262) Can someone tell me why hftp is failing to serve files, or at least where t= o look? --_000_9240AD66F4BBBA4088ACAAC15099237B7E4791CH1PRD0811MB430na_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks for the reply, Arp= it.

 <= /p>

Yes, both of those produc= e a correct response, although the second one’s syntax is:=

 <= /p>

hadoop fs –cat logf= iles/day_id=3D19991231/hour_id=3D1999123123/000008_0

 <= /p>

 <= /p>

From: Arpit Gu= pta [mailto:arpit@hortonworks.com]
Sent: Tuesday, December 18, 2012 3:49 PM

Hi Robert

 

Does the cat work for you if you dont use hftp, some= thing like

 

 

Or

 

 

 

--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/<= /span>

 

On Dec 18, 2012, at 2:43 PM, Robert Rapplean <robert.rapplean@trueffect.com= > wrote:



Hey, everone. Just go= t finished reading about all of the unsubscribe messages in Sept-Oct, and w= as hoping someone has a clue about what my system is doing wrong. I suspect= that this is a configuration issue, but I don't even know where to start looking for it. I'm a developer, and = my sysadmin is tied up until the end of the year.

I'm trying to move files from one cluster to another using distcp, using th= e hftp protocol as specified in their instructions.

I can read directories over hftp, but when I attempt to get a file I get a = 500 (internal server error). To eliminate the possibility of network and fi= rewall issues, I'm using hadoop fs -ls and hadoop fs -cat commands on the s= ource server in order to attempt to figure out this issue.

This provides a directory of the files, which is correct.

hadoop fs -ls ourlogs/day_id=3D19991231/hour_id=3D1999123123
-rw-r--r--   3 username supergroup      =   812 2012-12-16 17:21 logfiles/day_id=3D19991231/hour_id=3D19991= 23123/000008_0

This gives me a "file not found" error, which is also correct bec= ause the file isn't there:

hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D19991231= /hour_id=3D1999123123/000008_0x
cat: `hftp://hdenn00.trueffect= .com:50070/user/prodman/ods_fail/day_id=3D19991231/hour_id=3D1999123123/000= 008_0x': No such file or directory

This line gives me a 500 internal server error. The file is confirmed to be= on the server.

hadoop fs -cat hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D19991231= /hour_id=3D1999123123/000008_0
cat: HTTP_OK expected, received 500

Here is a stack trace of what distcp logs when I attempt this:

java.io.IOException: HTTP_OK expected, received 500
   at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOp= ener.connect(HftpFileSystem.java:365)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputS= tream(ByteRangeInputStream.java:119)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputSt= ream(ByteRangeInputStream.java:103)
   at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteR= angeInputStream.java:187)
   at java.io.DataInputStream.read(DataInputStream.java:83)<= br>    at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(Di= stCp.java:424)
   at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(Dis= tCp.java:547)
   at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(Dis= tCp.java:314)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:= 50)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.= java:393)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)=
   at org.apache.hadoop.mapred.Child$4.run(Child.java:268)    at java.security.AccessController.doPrivileged(Native Met= hod)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(U= serGroupInformation.java:1332)
   at org.apache.hadoop.mapred.Child.main(Child.java:262)
Can someone tell me why hftp is failing to serve files, or at least where t= o look?

 

--_000_9240AD66F4BBBA4088ACAAC15099237B7E4791CH1PRD0811MB430na_--