Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
  b=XadWKLsdRmEV56BCqXvyYq2wLw7hcEfrYhhA2n1Oh0oG2NeDOneGsepHTTezXd1s4kK3xwdOXf8djlr/zgbz4jOgbR9sMhUucbkWJJEnSTdlel/bpmhTbFQvSXicxQeDSZ2jNTbzhji5jRe3uoa6Ql1j2sVmO6SF+6lLyqs5Kvo=;
Message-ID: <1327824400.90923.YahooMailNeo@web164516.mail.gq1.yahoo.com>
Date: Sun, 29 Jan 2012 00:06:40 -0800 (PST)
From: Jianhui Zhang <wuhuagua@yahoo.com>
Reply-To: Jianhui Zhang <wuhuagua@yahoo.com>
Subject: anyway to do "local" reduce like the combiner does?
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1701424453-326791085-1327824400=:90923"

--1701424453-326791085-1327824400=:90923
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

=0A=0AI have a problem at hand that seems to need "local" reducing: =0A=0A=
=0AI have a large data input, in which each line is a data mapping, somethi=
ng like "name : attribute". The attributes for the same name are usually pr=
etty close in the file, so they are very likely to be processed by the same=
 mapper. I need to persist the "name:attributes" somewhere else (think DB).=
 It'll be optimal if I can combine the attributes of the same name together=
 and only persist them once. Attributes for the same name from different ma=
ppers can be safely persisted separately.=A0=0A=0AI don't want to use reduc=
ers due to the network traffic. What I need is exactly what a combiner does=
, but as far as I can tell, combiners are not guaranteed to run or run only=
 once (Correct me if I'm wrong here), so I guess I am not supposed to imple=
ment the persistence in the combiner.=A0=0A=0AAnybody has got a similar pro=
blem before? What's your solution? =0A=0A=0AAppreciate your help. =0A=0A=0A=
Thanks,=0AJames=0A
--1701424453-326791085-1327824400=:90923
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"color:#000; background-color:#fff; font-family:ar=
ial, helvetica, sans-serif;font-size:10pt"><div><br></div><div>I have a pro=
blem at hand that seems to need "local" reducing: <br></div><div><br></div>=
<div>I have a large data input, in which each line is a data mapping, somet=
hing like "name : attribute". The attributes for the same name are usually =
pretty close in the file, so they are very likely to be processed by the sa=
me mapper. I need to persist the "name:attributes" somewhere else (think DB=
). It'll be optimal if I can combine the attributes of the same name togeth=
er and only persist them once. Attributes for the same name from different =
mappers can be safely persisted separately.&nbsp;</div><div><br></div><div>=
I don't want to use reducers due to the network traffic. What I need is exa=
ctly what a combiner does, but as far as I can tell, combiners are not guar=
anteed to run or run only once (Correct me if I'm wrong here), so I
 guess I am not supposed to implement the persistence in the combiner.&nbsp=
;</div><div><br></div><div>Anybody has got a similar problem before? What's=
 your solution? <br></div><div><br></div><div>Appreciate your help. <br></d=
iv><div><br></div><div>Thanks,</div><div>James</div><div><br> </div></div><=
/body></html>
--1701424453-326791085-1327824400=:90923--