hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Einar Vollset" <ei...@somethingsimpler.com>
Subject distcp/ls fails on Hadoop-0.17.0 on ec2.
Date Thu, 29 May 2008 21:15:15 GMT
Hi,

I'm using the current Hadoop ec2 image (ami-ee53b687), and am having
some trouble getting hadoop
to access S3. Specifically, I'm trying to copy files from my bucket,
into HDFS on the running cluster, so
(on the master on the booted cluster) I do:

hadoop-0.17.0 einar$ bin/hadoop distcp
s3://ID:SECRET@my-test-bucket-with-alot-of-data/ input
08/05/29 14:10:44 INFO util.CopyFiles: srcPaths=[
s3://ID:SECRET@my-test-bucket-with-alot-of-data/]
08/05/29 14:10:44 INFO util.CopyFiles: destPath=input
08/05/29 14:10:46 WARN fs.FileSystem: "localhost:9000" is a deprecated
filesystem name. Use "hdfs://localhost:9000/" instead.
With failures, global counters are inaccurate; consider running with -i
Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input
source  s3://ID:SECRET@my-test-bucket-with-alot-of-data/ does not
exist.
        at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578)
        at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594)
        at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763)

..which clearly doesn't work. The ID:SECRET are right - as if I change
them I get :

org.jets3t.service.S3ServiceException: S3 HEAD request failed.
ResponseCode=403, ResponseMessage=Forbidden
..etc

I suspect it might be a generic problem, as if I do:

bin/hadoop fs -ls  s3://ID:SECRET@my-test-bucket-with-alot-of-data/

I get:
ls: Cannot access s3://ID:SECRET@my-test-bucket-with-alot-of-data/ :
No such file or directory.


..even though the bucket is there and has a lot of data in it.


Any thoughts?

Cheers,

Einar

Mime
View raw message