Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
From: Surbhi Mungre <mungre.surbhi@gmail.com>
Date: Wed, 2 Sep 2015 16:17:54 -0500
Message-ID: 
 <CAMZUPzDGsaZmJe6jgOsBuG8zKCNs__c-JrVX6e2d1g+HsBh3SQ@mail.gmail.com>
Subject: JoinStrategy with Spark pipeline
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a114088905156a3051eca347d

--001a114088905156a3051eca347d
Content-Type: text/plain; charset=UTF-8

I was trying to determine effect of changing JoinStrategy on a Spark
pipeline. I noticed that my pipeline works fine with DefaultJoinStrategy,
however I could not get it to working with MapSideJoinStrategy and
BloomFilterJoinStrategy. For MapSideJoinStrategy I get an exceptions[1] on
driver itself and for BloomFilterJoinStrategy I get exceptions[2] in one of
the stages. I have not tried to do any configuration changes but I did run
tests with datasets of different sizes to ensure that my PCollection is
small enough to fit in memory. I am running spark in yarn-client mode with
Crunch 0.11.0-cdh5.4.2.

[1] https://gist.github.com/anonymous/15d6c691b743ad392d42
[2] https://gist.github.com/anonymous/b02a82401a30a69f1cff

Thanks,
Surbhi

--001a114088905156a3051eca347d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I was trying to determine effect of changing JoinStrategy =
on a Spark pipeline. I noticed that my pipeline works fine with DefaultJoin=
Strategy, however I could not get it to working with MapSideJoinStrategy an=
d BloomFilterJoinStrategy. For=C2=A0MapSideJoinStrategy I get an exceptions=
[1] on driver itself and for BloomFilterJoinStrategy I get exceptions[2] in=
 one of the stages. I have not tried to do any configuration changes but I =
did run tests with datasets of different sizes to ensure that my PCollectio=
n is small enough to fit in memory. I am running spark in yarn-client mode =
with Crunch=C2=A00.11.0-cdh5.4.2.<div><br></div><div><div>[1]=C2=A0<a href=
=3D"https://gist.github.com/anonymous/15d6c691b743ad392d42" target=3D"_blan=
k">https://gist.github.com/anonymous/15d6c691b743ad392d42</a></div><div>[2]=
=C2=A0<a href=3D"https://gist.github.com/anonymous/b02a82401a30a69f1cff" ta=
rget=3D"_blank">https://gist.github.com/anonymous/b02a82401a30a69f1cff</a><=
/div><div><br></div><div>Thanks,</div><div>Surbhi</div></div></div>

--001a114088905156a3051eca347d--