hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject Re: When exactly is combiner invoked?
Date Wed, 27 Jan 2010 18:37:58 GMT
Hi,
To elaborate a little on Gang's point, the buffer threshold is limited by io.sort.spill.percent,
during which spills are created. If the number of spills is more than min.num.spills.for.combine,
combiner gets invoked on the spills created before writing to disk.
I'm not sure what exactly you intend to say by "finish processing an input record". Typically,
the processing (map) ends with a output.collect.

Amogh



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message