hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Yehdego <dtyehd...@miners.utep.edu>
Subject RE: Hadoop-streaming using binary executable c program
Date Fri, 22 Jul 2011 17:31:42 GMT

Hi Bobby, Thanks for the response.

After I tried the following comannd:

bin/hadoop jar $HADOOP_HOME/hadoop-0.20.2-streaming.jar -mapper /data/yehdego/hadoop-0.20.2/pknotsRG-1.3/src/pknotsRG
-  -file /data/yehdego/hadoop-0.20.2/pknotsRG-1.3/src/pknotsRG  -reducer NONE -input /user/yehdego/RNAData/RF00028_B.bpseqL3G5_seg_Centered_Method.txt
-output /user/yehdego/RF-out - verbose

I got a stderr logs :

java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 139
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)



syslog logs

2011-07-22 13:02:27,467 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
with processName=MAP, sessionId=
2011-07-22 13:02:27,913 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2011-07-22 13:02:28,149 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/data/yehdego/hadoop_tmp/dfs/local/taskTracker/jobcache/job_201107181535_0079/attempt_201107181535_0079_m_000000_0/work/./pknotsRG]
2011-07-22 13:02:28,242 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s]
out:NA [rec/s]
2011-07-22 13:02:28,267 INFO org.apache.hadoop.streaming.PipeMapRed: MROutputThread done
2011-07-22 13:02:28,267 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
2011-07-22 13:02:28,267 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!
2011-07-22 13:02:28,361 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 139
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
2011-07-22 13:02:28,395 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the
task



Regards, 

Daniel T. Yehdego
Computational Science Program 
University of Texas at El Paso, UTEP 
dtyehdego@miners.utep.edu

> From: evans@yahoo-inc.com
> To: common-user@hadoop.apache.org; dtyehdego@miners.utep.edu
> Date: Fri, 22 Jul 2011 09:12:18 -0700
> Subject: Re: Hadoop-streaming using binary executable c program
> 
> It looks like it tried to run your program and the program exited with a 1 not a 0. 
What are the stderr logs like for the mappers that were launched, you should be able to access
them through the Web GUI?  You might want to add in some stderr log messages to you c program
too. To be able to debug how far along it is going before exiting.
> 
> --Bobby Evans
> 
> On 7/22/11 9:19 AM, "Daniel Yehdego" <dtyehdego@miners.utep.edu> wrote:
> 
> I am trying to parallelize some very long RNA sequence for the sake of
> predicting their RNA 2D structures. I am using a binary executable c
> program called pknotsRG as my mapper. I tried the following bin/hadoop
> command:
> 
> HADOOP_HOME$ bin/hadoop
> jar /data/yehdego/hadoop-0.20.2/hadoop-0.20.2-streaming.jar
> -mapper /data/yehdego/hadoop-0.20.2/pknotsRG
> -file /data/yehdego/hadoop-0.20.2/pknotsRG
> -input /user/yehdego/RF00028_B.bpseqL3G5_seg_Centered_Method.txt
> -output /user/yehdego/RF-out -reducer NONE -verbose
> 
> but i keep getting the following error message:
> 
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
> failed with code 1
>         at
> org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
>         at
> org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
>         at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
>         at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>         at org.apache.hadoop.mapred.Child.main(Child.java:170)
> 
> FYI: my input file is RF00028_B.bpseqL3G5_seg_Centered_Method.txt which
> is a chunk of RNA sequences and the mapper is expected to get the input
> and execute the input file line by line and out put the predicted
> structure for each line of sequence for a specified number of maps. Any
> help on this problem is really appreciated. Thanks.
> 
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message