hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parth Savani <pa...@sensenetworks.com>
Subject problem using s3 instead of hdfs
Date Mon, 15 Oct 2012 17:13:14 GMT
Hello,
      I am trying to run hadoop on s3 using distributed mode. However I am
having issues running my job successfully on it. I get the following error
I followed the instructions provided in this article ->
http://wiki.apache.org/hadoop/AmazonS3
I replaced the fs.default.name value in my hdfs-site.xml to
s3n://ID:SECRET@BUCKET
And I am running my job using the following: hadoop jar
/path/to/my/jar/abcd.jar /input /output
Where */input* is the folder name inside the s3 bucket
(s3n://ID:SECRET@BUCKET/input)
and */output *folder should created in my bucket (s3n://ID:SECRET@BUCKET
/output)
Below is the error i get. It is looking for job.jar on s3 and that path is
on my server from where i am launching my job.

java.io.FileNotFoundException: No such file or directory
'/opt/data/hadoop/hadoop-mapred/mapred/staging/psavani/.staging/job_201207021606_1036/job.jar'
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:412)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1371)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1352)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:222)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1372)
at java.security.AccessController.doPri

Mime
View raw message