Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=E/L01KlMSMbik+yhGZS7Wj+/7OTragg9cyOQebRPs3GnwuZxDkveVu5lZf+kylUo7J5CadbijWgxxJqdz19dLwbkWIgaz5Kd5dFRGSdWCV501RA1Gfjlefk/G8Tf5pSyTtjj7Y9Ka+UTlZM97mQHDUGQUEQICZBpYeJ1uv0sSbM=;
Message-ID: <148081.21313.qm@web15903.mail.cnb.yahoo.com>
References: <85865e431001051312l2a0649a7i5ff3c3a2c4efbe53@mail.gmail.com>
  <662493.64083.qm@web15908.mail.cnb.yahoo.com>
 <ce6ab8461001051743x5be7f72fo44e4e9d6dae4f687@mail.gmail.com>
Date: Wed, 6 Jan 2010 13:55:32 +0800 (CST)
From: Gang Luo <lgpublic@yahoo.com.cn>
Subject: Re: combiner statistics
To: common-user@hadoop.apache.org
In-Reply-To: <ce6ab8461001051743x5be7f72fo44e4e9d6dae4f687@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: quoted-printable

Thanks. What I mean is, the combiner doesn't "intentionally" re-read spille=
d records back to memory just to combine them. But it does happens that som=
e records will be re-read for sort. I think combiner should work on those r=
ecords.=0A=0A =0A-Gang=0A=0A=0A=0A----- =D4=AD=CA=BC=D3=CA=BC=FE ----=0A=B7=
=A2=BC=FE=C8=CB=A3=BA Ted Xu <ted.xu.ml@gmail.com>=0A=CA=D5=BC=FE=C8=CB=A3=
=BA common-user@hadoop.apache.org=0A=B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/1/5=
 (=D6=DC=B6=FE) 8:43:53 =CF=C2=CE=E7=0A=D6=F7   =CC=E2=A3=BA Re: combiner s=
tatistics=0A=0AHi Gang,=0A=0AMy understanding to this is that, the combiner=
 has to re-read some records=0A> which have already been spilled to disk an=
d combine them with those records=0A> which come later.=0A>=0A=0AI believe =
the combine operation is done before map spill and after reduce=0Amerge. Co=
mbine only occurs in the memory, instead of re-read records from=0Adisks.=
=0A=0A=0A> Besides, I am not sure whether the combiner can guarantee there =
is only one=0A> record for each distinct key in each map task. Or does it j=
ust "try its=0A> best" to combine?=0A>=0A=0AYes, they can only "try their b=
est".=0A=0A=0A=0A      ____________________________________________________=
_______ =0A  =BA=C3=CD=E6=BA=D8=BF=A8=B5=C8=C4=E3=B7=A2=A3=AC=D3=CA=CF=E4=
=BA=D8=BF=A8=C8=AB=D0=C2=C9=CF=CF=DF=A3=A1 =0Ahttp://card.mail.cn.yahoo.com=
/