Return-Path: 
 <hdfs-user-return-425-apmail-hadoop-hdfs-user-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Received: (qmail 63717 invoked from network); 15 Jun 2010 15:52:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 15:52:29 -0000
Received: (qmail 7067 invoked by uid 500); 15 Jun 2010 15:52:29 -0000
Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org
Received: (qmail 6882 invoked by uid 500); 15 Jun 2010 15:52:28 -0000
Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-user@hadoop.apache.org>
List-Id: <hdfs-user.hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Delivered-To: mailing list hdfs-user@hadoop.apache.org
Received: (qmail 6874 invoked by uid 99); 15 Jun 2010 15:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 15:52:28 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com)
 (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 15:52:23 +0000
Received: by gxk8 with SMTP id 8so1376353gxk.35
        for <hdfs-user@hadoop.apache.org>;
 Tue, 15 Jun 2010 08:52:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.244.2 with SMTP id r2mr6241194anh.148.1276617119566; Tue,
	15 Jun 2010 08:51:59 -0700 (PDT)
Received: by 10.231.156.139 with HTTP; Tue, 15 Jun 2010 08:51:59 -0700 (PDT)
X-Originating-IP: [208.252.10.190]
In-Reply-To: <AANLkTimxehFdLl3810LMF-IcxwCN_-HG88d_qVF4lao4@mail.gmail.com>
References: <AANLkTimdENhE7r1VDSk4KkunWvT3hzkq63ZKrE8pZZgR@mail.gmail.com>
	<AANLkTimxehFdLl3810LMF-IcxwCN_-HG88d_qVF4lao4@mail.gmail.com>
Date: Tue, 15 Jun 2010 11:51:59 -0400
Message-ID: <AANLkTikKaVGYWyYy_6ePU4whxmkd27TMxFV_uFIDKC5T@mail.gmail.com>
Subject: Re: Using S3 native filesystem? (s3n://)
From: Denis Haskin <denis@haskinferguson.net>
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Actually, it still seems that "hadoop fs" is incorrectly using
core-site.xml instead of hdfs-site.xml.  If I put my fs.default.name &
etc only in hdfs-site.xml, "hadoop fs" doesn't find it.  But does if
it's in core-site.xml.

I assume this is a bug?

dwh



On Fri, Jun 11, 2010 at 6:30 AM, Denis Haskin <denis@haskinferguson.net> wr=
ote:
> Resolved this; http://wiki.apache.org/hadoop/AmazonS3 is a little unclear=
.
>
> The page says "Note that since the secret access key can contain
> slashes, you must remember to escape them by replacing each slash /
> with the string %2F."
>
> This appears to apply ONLY when the secret access key is embedded in
> the fs.default.name, not when it's specified separately via
> fs.s3n.awsSecretAccessKey. =A0I haven't fully tested that, but once I
> un-escaped it (in the awsSecretAccessKey), it works.
>
> Also, I don't know why I was trying to do a "namenode -format" in this
> case; it's not necesssary.
>
> dwh
>
>
>
> On Thu, Jun 10, 2010 at 2:15 PM, Denis Haskin <denis@haskinferguson.net> =
wrote:
>> I'm having trouble getting the S3 native filesystem working, trying to u=
se
>> it from my local desktop. =A0Following http://wiki.apache.org/hadoop/Ama=
zonS3,
>> I have my hdfs-site.xml as:
>> <configuration>
>> =A0=A0<property>
>> =A0=A0 =A0<name>dfs.replication</name>
>> =A0=A0 =A0<value>1</value>
>> =A0=A0</property>
>> <property>
>> =A0=A0<name>fs.default.name</name>
>> =A0=A0<value>s3n://dwh-hdfs-test</value>
>> </property>
>> <property>
>> =A0=A0<name>fs.s3n.awsAccessKeyId</name>
>> =A0=A0<value>..access key...</value>
>> </property>
>> <property>
>> =A0=A0<name>fs.s3n.awsSecretAccessKey</name>
>> =A0=A0<value>...secret access key, with / encoded as %2F..</value>
>> </property>
>> </configuration>
>> "bin/hadoop namenode -format" works; I end up with this structure (as se=
en
>> by s3cmd):
>> 2010-06-10 13:11 =A0 =A0 =A0 =A0 4
>> s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred/system/jobtracker.info
>> 2010-06-10 13:11 =A0 =A0 =A0 =A0 0
>> s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred/system_$folder$
>> 2010-06-10 13:10 =A0 =A0 =A0 =A0 0
>> s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred_$folder$
>> 2010-06-10 13:10 =A0 =A0 =A0 =A0 0 =A0 s3://dwh-hdfs-test/tmp/hadoop-dha=
skin_$folder$
>> 2010-06-10 13:10 =A0 =A0 =A0 =A0 0 =A0 s3://dwh-hdfs-test/tmp_$folder$
>> but when I try something like "bin/hadoop fs -ls" I get various errors.
>> =A0Part of the problem is it's sort of unclear what the format of the
>> parameter should be. =A0See what I tried and the various errors down bel=
ow, at
>> [1]
>> Also, when I started up the jobtracker, it fails with:
>> 2010-06-10 13:34:08,521 FATAL org.apache.hadoop.mapred.JobTracker:
>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceExcepti=
on:
>> S3 HEAD request failed for '/tmp%2Fhadoop-dhaskin%2Fmapred%2Fsystem' -
>> ResponseCode=3D403, ResponseMessage=3DForbidden
>> Suggestions? =A0Thanks...
>> dwh
>>
>> [1] bin/hadoop fs -ls -- various attempts:
>> dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls /
>> ls: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Err=
or
>> Message: <?xml version=3D"1.0"
>> encoding=3D"UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>Th=
e
>> request signature we calculated does not match the signature you provide=
d.
>> Check your key and signing method.</Message><StringToSignBytes>47 45 54 =
0a
>> 0a 0a 54 68 75 2c 20 31 30 20 4a 75 6e 20 32 30 31 30 20 31 37 3a 33 30 =
3a
>> 34 30 20 47 4d 54 0a 2f 64 77 68 2d 68 64 66 73 2d 74 65 73 74
>> 2f</StringToSignBytes><RequestId>098E7E853DB93FD9</RequestId><HostId>Y8w=
3ccqhj4VvWY6Ma17o5HF+8cWK3r1kiIywxAsrwSsuR2DqFpxx4+2+9Xgnn+9i</HostId><Sign=
atureProvided>w8vS4vJjm5MpkLpDbnuVcByruxw=3D</SignatureProvided><StringToSi=
gn>GETThu,
>> 10 Jun 2010 17:30:40
>> GMT/dwh-hdfs-test/</StringToSign><AWSAccessKeyId>AKIAI5UTNXFARAYFZAXQ</A=
WSAccessKeyId></Error>
>> dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test
>> ls: Path must be absolute: s3n://dwh-hdfs-test
>> Usage: java FsShell [-ls <path>]
>> dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test/
>> ls: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Err=
or
>> Message: <?xml version=3D"1.0"
>> encoding=3D"UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>Th=
e
>> request signature we calculated does not match the signature you provide=
d.
>> Check your key and signing method.</Message><StringToSignBytes>47 45 54 =
0a
>> 0a 0a 54 68 75 2c 20 31 30 20 4a 75 6e 20 32 30 31 30 20 31 37 3a 33 30 =
3a
>> 35 36 20 47 4d 54 0a 2f 64 77 68 2d 68 64 66 73 2d 74 65 73 74
>> 2f</StringToSignBytes><RequestId>2F7004189009A56A</RequestId><HostId>IQz=
sA5849ZnBaqGuAsFvTIt78u9oDRaBvrY5Xwg5exf85H+7/aAejxK33QPLXCue</HostId><Sign=
atureProvided>ryv+zgWT3YXPbqPcMNr0F4dcWKM=3D</SignatureProvided><StringToSi=
gn>GETThu,
>> 10 Jun 2010 17:30:56
>> GMT/dwh-hdfs-test/</StringToSign><AWSAccessKeyId>AKIAI5UTNXFARAYFZAXQ</A=
WSAccessKeyId></Error>
>> dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test/tmp
>> ls: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/=
tmp'
>> - ResponseCode=3D403, ResponseMessage=3DForbidden
>> dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls /tmp
>> ls: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/=
tmp'
>> - ResponseCode=3D403, ResponseMessage=3DForbidden
>>
>