hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White" <tom.e.wh...@gmail.com>
Subject Re: distcp/ls fails on Hadoop-0.17.0 on ec2.
Date Sun, 01 Jun 2008 06:58:09 GMT
Hi Einar,

How did you put the data onto S3, using Hadoop's S3 FileSystem or
using other S3 tools? If it's the latter then it won't work as the s3
scheme is for Hadoop's block-based S3 storage. Native S3 support is
coming - see https://issues.apache.org/jira/browse/HADOOP-930, but
it's not integrated yet.

Tom

On Thu, May 29, 2008 at 10:15 PM, Einar Vollset
<einar@somethingsimpler.com> wrote:
> Hi,
>
> I'm using the current Hadoop ec2 image (ami-ee53b687), and am having
> some trouble getting hadoop
> to access S3. Specifically, I'm trying to copy files from my bucket,
> into HDFS on the running cluster, so
> (on the master on the booted cluster) I do:
>
> hadoop-0.17.0 einar$ bin/hadoop distcp
> s3://ID:SECRET@my-test-bucket-with-alot-of-data/ input
> 08/05/29 14:10:44 INFO util.CopyFiles: srcPaths=[
> s3://ID:SECRET@my-test-bucket-with-alot-of-data/]
> 08/05/29 14:10:44 INFO util.CopyFiles: destPath=input
> 08/05/29 14:10:46 WARN fs.FileSystem: "localhost:9000" is a deprecated
> filesystem name. Use "hdfs://localhost:9000/" instead.
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input
> source  s3://ID:SECRET@my-test-bucket-with-alot-of-data/ does not
> exist.
>        at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578)
>        at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594)
>        at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763)
>
> ..which clearly doesn't work. The ID:SECRET are right - as if I change
> them I get :
>
> org.jets3t.service.S3ServiceException: S3 HEAD request failed.
> ResponseCode=403, ResponseMessage=Forbidden
> ..etc
>
> I suspect it might be a generic problem, as if I do:
>
> bin/hadoop fs -ls  s3://ID:SECRET@my-test-bucket-with-alot-of-data/
>
> I get:
> ls: Cannot access s3://ID:SECRET@my-test-bucket-with-alot-of-data/ :
> No such file or directory.
>
>
> ..even though the bucket is there and has a lot of data in it.
>
>
> Any thoughts?
>
> Cheers,
>
> Einar
>

Mime
View raw message