Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D2067D728 for ; Tue, 18 Dec 2012 22:49:34 +0000 (UTC) Received: (qmail 186 invoked by uid 500); 18 Dec 2012 22:49:30 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 99743 invoked by uid 500); 18 Dec 2012 22:49:30 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 99735 invoked by uid 99); 18 Dec 2012 22:49:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 22:49:30 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arpit@hortonworks.com designates 209.85.220.47 as permitted sender) Received: from [209.85.220.47] (HELO mail-pa0-f47.google.com) (209.85.220.47) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 22:49:25 +0000 Received: by mail-pa0-f47.google.com with SMTP id fa10so855836pad.6 for ; Tue, 18 Dec 2012 14:49:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:content-type:message-id:mime-version:subject:date :references:to:in-reply-to:x-mailer:x-gm-message-state; bh=z4YhSYhmQUljZvYnNbZ4vV+wMmpRqmOqhNqLKPxBSqg=; b=oXHXpxsbf9aZxjOaXNaZ3wIjlv2gLwnweE+Hd/sJOP3wJNrajtNc56jRwSX5oXFKcV Nv9M/0uxL4RAKGzpjCTBuzsK3TV28M+d/X7ZIpr44Q9WAAHCK/uWN3v1GZ0jug/HYI+h AwzKqWjiR/qjC4B+b/6Kh/1mTFy3nyC0dDRNo7Chpq10oXlQ31ZRcXhSir1gz/OpP7pB qTCTD+Xz4b0xCKB6SSnkDJuIXigz0mOyVVPYRqU2vWNnnPwp72AjVsK4qwP/Dc1G6Zoz dHLJEt2WVA/6DB+jMwLeoUfW+qEe4qheTtrsBxoUgXyZGqs3TFOxxDlTwrwSIrnW0Oio VNOA== X-Received: by 10.68.238.106 with SMTP id vj10mr11815455pbc.40.1355870944633; Tue, 18 Dec 2012 14:49:04 -0800 (PST) Received: from [10.11.3.104] (host1.hortonworks.com. [70.35.59.2]) by mx.google.com with ESMTPS id kn3sm1866113pbc.3.2012.12.18.14.49.02 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 18 Dec 2012 14:49:03 -0800 (PST) From: Arpit Gupta Content-Type: multipart/alternative; boundary="Apple-Mail=_B132694C-38A6-4002-94CE-8289C2328A0E" Message-Id: <89F44B63-CE39-4D0C-A86B-D7E26641D672@hortonworks.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: hftp can list directories but won't send files Date: Tue, 18 Dec 2012 14:49:01 -0800 References: <9240AD66F4BBBA4088ACAAC15099237B7E4565@CH1PRD0811MB430.namprd08.prod.outlook.com> To: user@hadoop.apache.org In-Reply-To: <9240AD66F4BBBA4088ACAAC15099237B7E4565@CH1PRD0811MB430.namprd08.prod.outlook.com> X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQltkenRhwcVE9K7zMoBzcLY4Ig40v/oB8RboFZl7nQd4Ppj3DzN/jLbsZ8fd8DiuvreFdWW X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_B132694C-38A6-4002-94CE-8289C2328A0E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Robert Does the cat work for you if you dont use hftp, something like hadoop fs -cat = hdfs://hdenn00.trueffect.com:8020/user/username/logfiles/day_id=3D19991231= /hour_id=3D1999123123/000008_0x Or hadoop fs -cat = /user/username/logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0x -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Dec 18, 2012, at 2:43 PM, Robert Rapplean = wrote: > Hey, everone. Just got finished reading about all of the unsubscribe = messages in Sept-Oct, and was hoping someone has a clue about what my = system is doing wrong. I suspect that this is a configuration issue, but = I don't even know where to start looking for it. I'm a developer, and my = sysadmin is tied up until the end of the year. >=20 > I'm trying to move files from one cluster to another using distcp, = using the hftp protocol as specified in their instructions. >=20 > I can read directories over hftp, but when I attempt to get a file I = get a 500 (internal server error). To eliminate the possibility of = network and firewall issues, I'm using hadoop fs -ls and hadoop fs -cat = commands on the source server in order to attempt to figure out this = issue. >=20 > This provides a directory of the files, which is correct. >=20 > hadoop fs -ls ourlogs/day_id=3D19991231/hour_id=3D1999123123 > -rw-r--r-- 3 username supergroup 812 2012-12-16 17:21 = logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0 >=20 > This gives me a "file not found" error, which is also correct because = the file isn't there: >=20 > hadoop fs -cat = hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D1999123= 1/hour_id=3D1999123123/000008_0x > cat: = `hftp://hdenn00.trueffect.com:50070/user/prodman/ods_fail/day_id=3D1999123= 1/hour_id=3D1999123123/000008_0x': No such file or directory >=20 > This line gives me a 500 internal server error. The file is confirmed = to be on the server. >=20 > hadoop fs -cat = hftp://hdenn00.trueffect.com:50070/user/username/logfiles/day_id=3D1999123= 1/hour_id=3D1999123123/000008_0 > cat: HTTP_OK expected, received 500 >=20 > Here is a stack trace of what distcp logs when I attempt this: >=20 > java.io.IOException: HTTP_OK expected, received 500 > at = org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFil= eSystem.java:365) > at = org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInput= Stream.java:119) > at = org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputS= tream.java:103) > at = org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java= :187) > at java.io.DataInputStream.read(DataInputStream.java:83) > at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424) > at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547) > at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1332) > at org.apache.hadoop.mapred.Child.main(Child.java:262) >=20 > Can someone tell me why hftp is failing to serve files, or at least = where to look? >=20 --Apple-Mail=_B132694C-38A6-4002-94CE-8289C2328A0E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi = Robert

Does the cat work for you if you dont use = hftp, something like





--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
<= /span>

On Dec 18, 2012, at 2:43 PM, Robert Rapplean <robert.rapplean@trueffect.co= m> wrote:

Hey, everone. Just got finished reading about all of the = unsubscribe messages in Sept-Oct, and was hoping someone has a clue = about what my system is doing wrong. I suspect that this is a = configuration issue, but I don't even know where to start looking for = it. I'm a developer, and my sysadmin is tied up until the end of the = year.

I'm trying to move files from one cluster to another using = distcp, using the hftp protocol as specified in their = instructions.

I can read directories over hftp, but when I = attempt to get a file I get a 500 (internal server error). To eliminate = the possibility of network and firewall issues, I'm using hadoop fs -ls = and hadoop fs -cat commands on the source server in order to attempt to = figure out this issue.

This provides a directory of the files, = which is correct.

hadoop fs -ls = ourlogs/day_id=3D19991231/hour_id=3D1999123123
-rw-r--r-- =   3 username supergroup =        812 2012-12-16 17:21 = logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0

This = gives me a "file not found" error, which is also correct because the = file isn't there:

hadoop fs -cat hftp://hdenn00.trueffect.com:5007= 0/user/username/logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0x<= /a>
cat: `hftp://hdenn00.trueffect.com:50= 070/user/prodman/ods_fail/day_id=3D19991231/hour_id=3D1999123123/000008_0x= ': No such file or directory

This line gives me a 500 = internal server error. The file is confirmed to be on the = server.

hadoop fs -cat hftp://hdenn00.trueffect.com:50070= /user/username/logfiles/day_id=3D19991231/hour_id=3D1999123123/000008_0
cat: HTTP_OK expected, received 500

Here is a stack trace of = what distcp logs when I attempt this:

java.io.IOException: = HTTP_OK expected, received 500
   at = org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFil= eSystem.java:365)
   at = org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInput= Stream.java:119)
   at = org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputS= tream.java:103)
   at = org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java= :187)
   at = java.io.DataInputStream.read(DataInputStream.java:83)
=    at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424)
=    at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547)
=    at = org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314)
=    at = org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
=    at = org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
=    at = org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
=    at = org.apache.hadoop.mapred.Child$4.run(Child.java:268)
=    at java.security.AccessController.doPrivileged(Native = Method)
   at = javax.security.auth.Subject.doAs(Subject.java:396)
=    at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1332)
   at = org.apache.hadoop.mapred.Child.main(Child.java:262)

Can someone = tell me why hftp is failing to serve files, or at least where to = look?


= --Apple-Mail=_B132694C-38A6-4002-94CE-8289C2328A0E--