hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasan Ary <voicesnthed...@yahoo.com>
Subject Re: distcp fails :Input source not found
Date Wed, 02 Apr 2008 14:41:50 GMT
Anybody ? Any thoughts why this might be happening?
   
  Here is what is happening directly from the ec2 screen. The ID and
 Secret Key are the only things changed.
  
  I'm running hadoop 15.3 from the public ami. I launched a 2 machine
 cluster using the ec2 scripts in  the src/contrib/ec2/bin . . .

    The file I try and copy is 9KB (I noticed previous discussion on
 empty files and files that are > 10MB)
   
  >>>>> First I make sure that we can copy the file from s3
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# bin/hadoop fs
 -copyToLocal s3://ID:SecretKey@randbucket123/InputFileFormat.xml
 /usr/InputFileFormat.xml
 
   >>>>> Now I see that the file is copied to the ec2 master (where I'm
 logged in)
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# dir /usr/Input*
  /usr/InputFileFormat.xml
   
  >>>>> Next I make sure I can access the HDFS and that the input
 directory is there
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# bin/hadoop fs -ls /
  Found 2 items
  /input <dir> 2008-04-01 15:45
  /mnt <dir> 2008-04-01 15:42
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# bin/hadoop fs -ls
 /input/
  Found 0 items
   
  >>>>> I make sure hadoop is running just fine by running an example
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# bin/hadoop jar
 hadoop-0.15.3-examples.jar pi 10 1000
  Number of Maps = 10 Samples per Map = 1000
  Wrote input for Map #0
  Wrote input for Map #1
  Wrote input for Map #2
  Wrote input for Map #3
  Wrote input for Map #4
  Wrote input for Map #5
  Wrote input for Map #6
  Wrote input for Map #7
  Wrote input for Map #8
  Wrote input for Map #9
  Starting Job
  08/04/01 17:38:14 INFO mapred.FileInputFormat: Total input paths to
 process : 10
  08/04/01 17:38:14 INFO mapred.JobClient: Running job:
 job_200804011542_0001
  08/04/01 17:38:15 INFO mapred.JobClient: map 0% reduce 0%
  08/04/01 17:38:22 INFO mapred.JobClient: map 20% reduce 0%
  08/04/01 17:38:24 INFO mapred.JobClient: map 30% reduce 0%
  08/04/01 17:38:25 INFO mapred.JobClient: map 40% reduce 0%
  08/04/01 17:38:27 INFO mapred.JobClient: map 50% reduce 0%
  08/04/01 17:38:28 INFO mapred.JobClient: map 60% reduce 0%
  08/04/01 17:38:31 INFO mapred.JobClient: map 80% reduce 0%
  08/04/01 17:38:33 INFO mapred.JobClient: map 90% reduce 0%
  08/04/01 17:38:34 INFO mapred.JobClient: map 100% reduce 0%
  08/04/01 17:38:43 INFO mapred.JobClient: map 100% reduce 20%
  08/04/01 17:38:44 INFO mapred.JobClient: map 100% reduce 100%
  08/04/01 17:38:45 INFO mapred.JobClient: Job complete:
 job_200804011542_0001
  08/04/01 17:38:45 INFO mapred.JobClient: Counters: 9
  08/04/01 17:38:45 INFO mapred.JobClient: Job Counters 
  08/04/01 17:38:45 INFO mapred.JobClient: Launched map tasks=10
  08/04/01 17:38:45 INFO mapred.JobClient: Launched reduce tasks=1
  08/04/01 17:38:45 INFO mapred.JobClient: Data-local map tasks=10
  08/04/01 17:38:45 INFO mapred.JobClient: Map-Reduce Framework
  08/04/01 17:38:45 INFO mapred.JobClient: Map input records=10
  08/04/01 17:38:45 INFO mapred.JobClient: Map output records=20
  08/04/01 17:38:45 INFO mapred.JobClient: Map input bytes=240
  08/04/01 17:38:45 INFO mapred.JobClient: Map output bytes=320
  08/04/01 17:38:45 INFO mapred.JobClient: Reduce input groups=2
  08/04/01 17:38:45 INFO mapred.JobClient: Reduce input records=20
  Job Finished in 31.028 seconds
  Estimated value of PI is 3.1556
   
  >>>>> Finally, I try and copy the file over
  [root@domU-12-31-39-00-5A-35 hadoop-0.15.3]# bin/hadoop distcp
 s3://ID:SecretKey@randbucket123/InputFileFormat.xml
 /input/InputFileFormat.xml
  With failures, global counters are inaccurate; consider running with
 -i
  Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input
 source s3://ID:SecretKey@randbucket123/InputFileFormat.xml does not
 exist.
  at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:470)
  at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:550)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
  at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:563)


       
---------------------------------
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No
Cost.
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message