hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddhartha Reddy" <s...@grok.in>
Subject distcp fails when copying from s3 to hdfs
Date Fri, 04 Apr 2008 08:32:33 GMT
I am trying to run a Hadoop cluster on Amazon EC2 and backup all the data on
Amazon S3 between the runs. I am using Hadoop 0.16.1 on a cluster made up of
CentOS 5 images (ami-08f41161).


I am able to copy from hdfs to S3 using the following command:

bin/hadoop distcp file.txt s3://id:secret@bucket-name/file.txt


But copying from S3 to hdfs with the following command fails:

bin/hadoop distcp s3://id:secret@bucket-name/file.txt file2.txt


with the following error:

With failures, global counters are inaccurate; consider running with -i
Copy failed: java.lang.IllegalArgumentException: Hook previously registered
    at
java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:45)
    at java.lang.Runtime.addShutdownHook(Runtime.java:192)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1194)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:148)
    at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:81)
    at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1180)
    at org.apache.hadoop.fs.FileSystem.access$400(FileSystem.java:53)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1197)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:148)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
    at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:482)
    at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:504)
    at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:580)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:596)


Can someone please point out if and what I am doing wrong?

Thanks,
Siddhartha Reddy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message