Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=KhoU9Sgi0sYVN16ej3uC2vLPij7SPEVbf5H/4nNocDpixgWTFzePECV9Tjlw56xNhl6yH5v/lEoCBKYwFv+ZuDZ0S97TSFUg9Hj3B7OpRqaRyqtHuLboX2rtqRBYPHOE6LqC7J2UvPKDJCx2JbuTYtFBXi5glQSLDeV0hUOcn5Q=;
Message-ID: <34630.84008.qm@web15902.mail.cnb.yahoo.com>
References: <4B607064.4000008@cs.cmu.edu>
Date: Thu, 28 Jan 2010 01:57:58 +0800 (CST)
From: Gang Luo <lgpublic@yahoo.com.cn>
Subject: Re: When exactly is combiner invoked?
To: common-user@hadoop.apache.org
In-Reply-To: <4B607064.4000008@cs.cmu.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: quoted-printable

When the map function generate the intermediate result and first sent them =
to buffer, the partitioning and sorting will start working and , if you spe=
cify a combiner, it will be invoked at this time. This process is in parall=
el with the map function. When map function finishes, all the spills on dis=
k will be merged, combiners will also be invoked at this time. =0A=0A-Gang=
=0A=0A=0A=0A----- =D4=AD=CA=BC=D3=CA=BC=FE ----=0A=B7=A2=BC=FE=C8=CB=A3=BA =
Le Zhao <lezhao@cs.cmu.edu>=0A=CA=D5=BC=FE=C8=CB=A3=BA common-user@hadoop.a=
pache.org=0A=B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/1/27 (=D6=DC=C8=FD) 11:57:0=
8 =C9=CF=CE=E7=0A=D6=F7   =CC=E2=A3=BA When exactly is combiner invoked?=0A=
=0AHi - combiner performs on a chunk of mapper output data, but what exactl=
y is the chunk cut off, or when exactly will the chunk be fed to the combin=
er?=0A=0A1. Will it be after the mapper finishes processing an input record=
?=0A2. Will it be after the mapper outputs a key value pair that hits the m=
emory limit?=0A=0AThis will be important to know, because strategy 1 gives =
more guarantee over output record duplicity than 2, say when an input recor=
d for the mapper can correspond to multiple output records with the same ke=
y.=0A=0AThanks,=0ALe=0A=0A=0A=0A      _____________________________________=
______________________ =0A  =BA=C3=CD=E6=BA=D8=BF=A8=B5=C8=C4=E3=B7=A2=A3=
=AC=D3=CA=CF=E4=BA=D8=BF=A8=C8=AB=D0=C2=C9=CF=CF=DF=A3=A1 =0Ahttp://card.ma=
il.cn.yahoo.com/