hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <omal...@apache.org>
Subject Re: setting a different input/output class for combiner function than map and reduce functions
Date Wed, 24 Sep 2008 16:09:53 GMT

On Sep 24, 2008, at 2:24 AM, Devaraj Das wrote:

> If you are on 0.18, it is possible to say that a combiner be invoked  
> once
> per partition per spill. Do
> job.setCombineOnlyOnce(true);

However, that functionality was introduced for backwards compatibility  
with versions prior to 0.18 and was removed from 0.19. The combiner  
should be viewed as a hint to framework for how to reduce the  
transient data size. Your application really shouldn't be doing  
transformations in the combiner.

-- Owen

View raw message