hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mu Qiao <qiao...@gmail.com>
Subject Combiner Problem
Date Mon, 06 Jul 2009 06:34:07 GMT
Hi, everyone

I've been learning hadoop recently and I'm confused about the combiner

There is a property min.num.spills.for.combine specifying the minimum number
of spills to run combiner when merging. The default value is 3. Why there is
such a restriction? Should it be better that run the combiner no matter how
many spills there are?

The second question is why the combiner could be run at the reduce side.
Can't the reduce function take place of that?

Thanks very much.

Best wishes,
Qiao Mu
MOE KLINNS Lab and SKLMS Lab, Xi'an Jiaotong University
Department of Computer Science and Technology, Xi’an Jiaotong University
TEL: 15991676983
E-mail: qiaomuf@gmail.com

View raw message