Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 27583 invoked from network); 2 Aug 2010 20:35:16 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Aug 2010 20:35:16 -0000 Received: (qmail 88154 invoked by uid 500); 2 Aug 2010 20:35:14 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 88125 invoked by uid 500); 2 Aug 2010 20:35:13 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 88117 invoked by uid 99); 2 Aug 2010 20:35:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:35:13 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:35:08 +0000 Received: by vws2 with SMTP id 2so3495696vws.35 for ; Mon, 02 Aug 2010 13:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=kjFumntkF9zymehahxEGFtslOXudDhbOOSrgWuiiC2E=; b=Fil0CLQzs6XVCyWYMNE+MexBkAZ7I4xBsTyaISJY+3KNVRVQGoGYzWO0DKZq8JADPK i3Xi0PZmlKwDeW7OyMdxgmoVYEugSesIkxjRqF5Z/FC8qDyg5igNZBQQBTq8IZ7GYAkZ lqMXKIHZKZlgy42rXN7XocXPCydJNIfeNVy0Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=X9VdTn52om9G/kZefVSh/QUT1+CN466784dQwWcE3b6FcSHCyPM+R7383IQ0txD4y8 WIQKDFQA+0e3w/9ANpa5VQhCyZQORVPygYlLfWuTxhzDq5x3oZNw2sF4ZvSzj7S5alLX RxsvppusXHbi0jrxpf4Znv78rIX7EB2WPfF4E= MIME-Version: 1.0 Received: by 10.220.88.147 with SMTP id a19mr4584790vcm.119.1280781286541; Mon, 02 Aug 2010 13:34:46 -0700 (PDT) Received: by 10.220.171.193 with HTTP; Mon, 2 Aug 2010 13:34:46 -0700 (PDT) In-Reply-To: References: <4C56E791.6060500@amd.com> Date: Mon, 2 Aug 2010 16:34:46 -0400 Message-ID: Subject: Re: Combiner function From: Edward Capriolo To: common-user@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Aug 2, 2010 at 4:28 PM, Jackob Carlsson wrote: > Thanks Nick, but "in-memory" means a combiner can only be used over a sin= gle > mapper?right?! Is there a way we use it for several mappers as well? Also > what do you mean by "it may or may not run on a particular map attempt"? > > Br, > Jackob > > On Mon, Aug 2, 2010 at 5:43 PM, Nick Jones wrote: > >> Hi Jackob, >> A combiner acts a lot like a reduce step but it's executed on the mapper >> with in-memory data. =A0I've seen a reduction in job execution time by a= dding >> one. =A0The one caveat to keep in mind is that it may or may not run on = a >> particular map attempt. >> >> Nick >> >> >> >> On 8/2/2010 10:39 AM, Jackob Carlsson wrote: >> >>> Hi everyone, >>> Could anyone please help me to understand the function of combiner? >>> >>> Thanks in advance >>> Jackob >>> >>> >> >> > > Is there a way we use it for several mappers as well? No. That is the exact opposite goal of the combiner. It runs locally. >it may or may not run on a particular map attempt It only runs when certain thresholds in the framework are reached. http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-t= o-use-or-not-to-use-a-combiner/