hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj Das <d...@yahoo-inc.com>
Subject Re: Reducer goes past 100% complete?
Date Tue, 10 Mar 2009 02:00:05 GMT
There is a jira for this https://issues.apache.org/jira/browse/HADOOP-5210. There was a jira
to address this problem with intermediate compression on and that's fixed - https://issues.apache.org/jira/browse/HADOOP-3131.


On 3/9/09 9:15 PM, "Doug Cook" <nabble@candiru.com> wrote:



Hi folks,

I've recently upgraded to Hadoop 0.19.1 from a much, much older version of
Hadoop.

Most things in my application (a highly modified version of Nutch) are
working just fine, but one of them is bombing out with odd symptoms. The map
works just fine, but then reduce phase (a) runs extremely slowly and (b) the
"percentage complete" reporting for each reduce task doesn't stop at 100%,
it just keeps going on past that.

I figure I'll start by understanding the percentage-complete reporting
issue, since it's pretty concrete and may have some bearing on the
performance issue. It seems likely that my application is mis-configuring
the job, or otherwise not correctly using the Hadoop API. I don't think I'm
doing anything way out of the ordinary; my reducer simply creates an object,
wraps it in an ObjectWritable, and calls output.collect(), and I have a
local class that implements OutputFormat to take the object and put it in a
Lucene index. It does actually create correct output, at least for small
indices; on large indices, the performance problems are killing me.

I can and will start rummaging around in the Hadoop code to figure out how
it calculates percentage complete, and see what I'm not doing correctly, but
thought I'd ask here, too, to see if someone has good suggestions off the
top of their head.

Many thanks-

Doug Cook
--
View this message in context: http://www.nabble.com/Reducer-goes-past-100--complete--tp22413589p22413589.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message