Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: <CAC79Lca0t_0yp1aKV_8Sszmy1Ym_D=uYiTOE6N=ZH70hiPjypg@mail.gmail.com>
References: <CANMpy95RZb0YxD96nuLdsCVb1y80M6mkZb=ebS5hi1F31XALuw@mail.gmail.com>
	<CAC79Lca0t_0yp1aKV_8Sszmy1Ym_D=uYiTOE6N=ZH70hiPjypg@mail.gmail.com>
Date: Wed, 11 May 2016 09:42:27 +0800
Message-ID: <CANMpy95SjViAtNhZ3e-yExb9DYBW3Nvh=ZLjt6-pGqj=fRUUVA@mail.gmail.com>
Subject: Re: confused about the MapsideJoinStrategy, why use
 LoadLeftSideMapsideJoinStrategy, what if left table is too large to store in memory?
From: =?UTF-8?B?6ZmI56ue?= <cj.magina@gmail.com>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a11c02dda7c201a053287278b
archived-at: Wed, 11 May 2016 01:42:40 -0000

--001a11c02dda7c201a053287278b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

mapsideJoinStrategy.create()  use LoadLeftSideMapsideJoinStrategy, i'm just
confused why LoadLeftSideMapsideJoinStrategy is better than default
strategy.

according to the annotation, LoadLeftSideMapsideJoinStrategy peforms better
than default strategy, but i don't know why

2016-05-10 11:30 GMT+08:00 David Ortiz <dpo5003@gmail.com>:

> Try mapsideJoinStrategy.create()
>
> On Mon, May 9, 2016, 9:29 PM =E9=99=88=E7=AB=9E <cj.magina@gmail.com> wro=
te:
>
>> hi, i'm very confused when i use MapsideJoinStrategy. the origin
>> constructor was deprecated, instead, LoadLeftSideMapsideJoinStrategy was
>> recommended, the main improvement is that load left side table in memory=
,
>> whose size is large than right side. however, when i want to use mas sid=
e
>> join, the left side table usually is too large to store in memory.
>>
>> for example i have to table A and B, we need A left join B, and
>> size(A)>>size(B), naturally we want to use map side join, and use A as l=
eft
>> side, B as right side, then load B in memory to process, it's very simpl=
e.
>> However, if we use LoadLeftSideMapsideJoinStrategy, we use A as right si=
de,
>> B as left side, which makes no improvement while adding a reverse DoFn
>>
>>
>>
>> --
>> =E9=99=88=E7=AB=9E=EF=BC=8C=E4=B8=AD=E7=A7=91=E9=99=A2=E8=AE=A1=E7=AE=97=
=E6=8A=80=E6=9C=AF=E7=A0=94=E7=A9=B6=E6=89=80=EF=BC=8C=E9=AB=98=E6=80=A7=E8=
=83=BD=E8=AE=A1=E7=AE=97=E6=9C=BA=E4=B8=AD=E5=BF=83
>> Jing Chen HPCC.ICT.AC China
>>
>


--=20
=E9=99=88=E7=AB=9E=EF=BC=8C=E4=B8=AD=E7=A7=91=E9=99=A2=E8=AE=A1=E7=AE=97=E6=
=8A=80=E6=9C=AF=E7=A0=94=E7=A9=B6=E6=89=80=EF=BC=8C=E9=AB=98=E6=80=A7=E8=83=
=BD=E8=AE=A1=E7=AE=97=E6=9C=BA=E4=B8=AD=E5=BF=83
Jing Chen HPCC.ICT.AC China

--001a11c02dda7c201a053287278b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><span style=3D"font-size:14px">mapsideJoinStrategy.create(=
) =C2=A0use=C2=A0</span><span style=3D"font-size:14px">LoadLeftSideMapsideJ=
oinStrateg</span><span style=3D"font-size:14px">y, i&#39;m just confused wh=
y=C2=A0</span><span style=3D"font-size:14px">LoadLeftSideMapsideJoinStrateg=
</span><span style=3D"font-size:14px">y is better than default strategy.</s=
pan><div><span style=3D"font-size:14px"><br></span><div><span style=3D"font=
-size:14px">according to the annotation,=C2=A0</span><span style=3D"font-si=
ze:14px">LoadLeftSideMapsideJoinStrateg</span><span style=3D"font-size:14px=
">y peforms better than default strategy, but i don&#39;t know why</span></=
div></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2=
016-05-10 11:30 GMT+08:00 David Ortiz <span dir=3D"ltr">&lt;<a href=3D"mail=
to:dpo5003@gmail.com" target=3D"_blank">dpo5003@gmail.com</a>&gt;</span>:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex"><p dir=3D"ltr">Try mapsideJoinStrategy.cre=
ate()<br>
</p><div class=3D"HOEnZb"><div class=3D"h5">
<br><div class=3D"gmail_quote"><div dir=3D"ltr">On Mon, May 9, 2016, 9:29 P=
M =E9=99=88=E7=AB=9E &lt;<a href=3D"mailto:cj.magina@gmail.com" target=3D"_=
blank">cj.magina@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex"><div dir=3D"ltr">hi, i&#39;m very confused when i use MapsideJoinStr=
ategy. the origin constructor was deprecated, instead,=C2=A0LoadLeftSideMap=
sideJoinStrategy was recommended, the main improvement is that load left si=
de table in memory, whose size is large than right side. however, when i wa=
nt to use mas side join, the left side table usually is too large to store =
in memory.=C2=A0<div><br></div><div>for example i have to table A and B, we=
 need A left join B, and size(A)&gt;&gt;size(B), naturally we want to use m=
ap side join, and use A as left side, B as right side, then load B in memor=
y to process, it&#39;s very simple. However, if we use=C2=A0LoadLeftSideMap=
sideJoinStrategy, we use A as right side, B as left side, which makes no im=
provement while adding a reverse DoFn=C2=A0</div></div><div dir=3D"ltr"><di=
v><br><div><div><br></div><div><br></div>-- <br><div>=E9=99=88=E7=AB=9E=EF=
=BC=8C=E4=B8=AD=E7=A7=91=E9=99=A2=E8=AE=A1=E7=AE=97=E6=8A=80=E6=9C=AF=E7=A0=
=94=E7=A9=B6=E6=89=80=EF=BC=8C=E9=AB=98=E6=80=A7=E8=83=BD=E8=AE=A1=E7=AE=97=
=E6=9C=BA=E4=B8=AD=E5=BF=83<br>Jing Chen <a href=3D"http://HPCC.ICT.AC" tar=
get=3D"_blank">HPCC.ICT.AC</a> China</div>
</div></div></div></blockquote></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div class=3D"gmail_signature">=E9=99=88=E7=AB=9E=EF=BC=8C=E4=B8=AD=E7=A7=
=91=E9=99=A2=E8=AE=A1=E7=AE=97=E6=8A=80=E6=9C=AF=E7=A0=94=E7=A9=B6=E6=89=80=
=EF=BC=8C=E9=AB=98=E6=80=A7=E8=83=BD=E8=AE=A1=E7=AE=97=E6=9C=BA=E4=B8=AD=E5=
=BF=83<br>Jing Chen <a href=3D"http://HPCC.ICT.AC" target=3D"_blank">HPCC.I=
CT.AC</a> China</div>
</div>

--001a11c02dda7c201a053287278b--