hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rasit OZDAS <rasitoz...@gmail.com>
Subject Re: Identify the input file for a failed mapper/reducer
Date Thu, 02 Apr 2009 15:10:43 GMT
Two quotes for this problem:

"Streaming map tasks should have a "map_input_file" environment
variable like the following:

"the value for map.input.file gives you the exact information you need."

(didn't try)

2009/3/26 Jason Fennell <jdfennell@gmail.com>:
> Is there a way to identify the input file a mapper was running on when
> it failed?  When a large job fails because of bad input lines I have
> to resort to rerunning the entire job to isolate a single bad line
> (since the log doesn't contain information on the file that that
> mapper was running on).
> Basically, I would like to be able to do one of the following:
> 1. Find the file that a mapper was running on when it failed
> 2. Find the block that a mapper was running on when it failed (and be
> able to find file names from block ids)
> I haven't been able to find any documentation on facilities to
> accomplish either (1) or (2), so I'm hoping someone on this list will
> have a suggestion.
> I am using the Hadoop streaming API on hadoop 0.18.2.
> -Jason

M. Raşit ÖZDAŞ

View raw message