hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wiley <kwi...@keithwiley.com>
Subject Re: C++ pipes on full (nonpseudo) cluster
Date Wed, 31 Mar 2010 13:57:49 GMT
On 2010, Mar 31, at 4:25 AM, Gianluigi Zanetti wrote:

> What happens if you try this:
>
> $ hadoop fs -rmr HDFSPATH/output ; hadoop pipes -D  
> hadoop.pipes.executable=EXECUTABLE -D  
> hadoop.pipes.java.recordreader=true -D  
> hadoop.pipes.java.recordwriter=true -input HDFSPATH/input -output  
> HDFSPATH/output


Not good news.  This is what I got:

$ hadoop pipes -D hadoop.pipes.executable=/Users/keithwiley/Astro_LSST/ 
hadoop-0.20.1+152/Mosaic/clue/Mosaic/src/cpp/Mosaic -D  
hadoop.pipes.java.recordreader=true -D  
hadoop.pipes.java.recordwriter=true -input /uwphysics/kwiley/mosaic/ 
input -output /uwphysics/kwiley/mosaic/output
Exception in thread "main" java.io.FileNotFoundException: File does  
not exist: /Users/keithwiley/Astro_LSST/hadoop-0.20.1+152/Mosaic/clue/ 
Mosaic/src/cpp/Mosaic
	at  
org 
.apache 
.hadoop 
.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java: 
457)
	at  
org 
.apache 
.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java: 
509)
	at  
org 
.apache 
.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:681)
	at  
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:802)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1290)
	at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
	at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
	at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)

Incidentally, just in case you're wondering:
$ ls -l /Users/keithwiley/Astro_LSST/hadoop-0.20.1+152/Mosaic/clue/ 
Mosaic/src/cpp/
total 800
368 -rwxr-xr-x  1 keithwiley  keithwiley  185184 Mar 29 19:08 Mosaic*
...other files...

The path is obviously correct on my local machine.  The only  
explanation is that Hadoop is looking for it on HDFS under that path.

I'm desperate.  I don't understand why I'm the only person who can get  
this working.  Could you please describe to me the set of commands you  
use to run a pipes program on a fully distributed cluster?

________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com     
music.keithwiley.com

"The easy confidence with which I know another man's religion is folly  
teaches
me to suspect that my own is also."
                                            --  Mark Twain
________________________________________________________________________________


Mime
View raw message