hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: Hadoop: Reduce exceeding 100% - a bug?
Date Thu, 09 Jul 2009 21:31:58 GMT
Reduce tasks which require more than twenty minutes are not a problem. But
you must emit some data periodically to inform the rest of the system that
each reducer is still alive. Emitting a (k, v) output pair to the collector
will reset the timer. Similarly, calling Reporter.incrCounter() will also
reset the clock. So if you're doing a large amount of processing in a loop
before you emit your final key value pairs, you should periodically
increment a counter to allow the rest of the system to confirm that you're
not deadlocked.

I'm not sure why your progress went so high. I know that Hadoop has some
quirks related to compression. If you've got compressed data, then
percentages might be inaccurate since the completed/available_input data
ratio will be partially based on compressed sizes.
- Aaron

On Thu, Jul 9, 2009 at 12:24 PM, Prashant Ullegaddi <
prashant.ullegaddi@research.iiit.ac.in> wrote:

> Hi Jothi,
>
> We are trying to index around 245GB compressed data (~1TB uncompressed)
> on a 9 node Hadoop cluster with 8 slaves and 1 master. In Map, we are
> just parsing the files, passing the same to reduce. In Reduce, we are
> indexing the parsed data in much like Nutch style.
>
> When we ran the job, map got over in less than 4hrs. But strange thing
> happened with reduces. They went past 100% progress (some 200%!). They
> showed 200+% before getting killed! Is this some kind of bug in Hadoop?
>
> All eventually got killed saying "Task
> attempt_200907091637_0004_r_000000_0 failed to report status for 1201
> seconds. Killing!" But I guess indexing in reduce takes more than 1200+
> seconds. How to go about it?
>
>
> Thanks in advance,
> Prashant,
> Search and Information Extraction Lab,
> IIIT-Hyderabad,
> INDIA.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message