hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew McNabb <amcn...@mcnabbs.org>
Subject Re: Streaming in Hadoop 0.10
Date Tue, 09 Jan 2007 19:36:10 GMT
On Mon, Jan 08, 2007 at 07:16:50PM -0700, Andrew McNabb wrote:
> In the past, I would get a job failure if a streaming mapper or reducer
> died (for example, due to a Python exception).  However, in 0.10, I've
> been getting output in the Hadoop task list that looks like:
> 
> tip_0002_m_000000	100.00%	MRErr: ImportError: No module named
> scipy

On the same note, I'm currently getting the following traceback in the
errors section of the map task list:

java.lang.NullPointerException
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:455)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:578)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:109)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:51)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:183)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1341)

The map task failed, and it was the mappers fault, but this error
message makes it appear that something in Hadoop was to blame.  However,
the real cause of failure is that the mapper raised a an AttributeError
and died.

Since it sounded like someone was working on improving the display of
error messages on the task list page, I thought this might be useful.

Thanks for everything.


-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868

Mime
View raw message