spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasun Ratn <prasun.r...@gmail.com>
Subject Re: trying to use Spark applications with modified Kryo
Date Mon, 17 Oct 2016 13:59:26 GMT
Thanks a lot Steve!

On Mon, Oct 17, 2016 at 4:59 PM, Steve Loughran <stevel@hortonworks.com> wrote:
>
> On 17 Oct 2016, at 10:02, Prasun Ratn <prasun.ratn@gmail.com> wrote:
>
> Hi
>
> I want to run some Spark applications with some changes in Kryo serializer.
>
> Please correct me, but I think I need to recompile spark (instead of
> just the Spark applications) in order to use the newly built Kryo
> serializer?
>
> I obtained Kryo 3.0.3 source and built it (mvn package install).
>
> Next, I took the source code for Spark 2.0.1 and built it (build/mvn
> -X -DskipTests -Dhadoop.version=2.6.0 clean package)
>
> I then compiled the Spark applications.
>
> However, I am not seeing my Kryo changes when I run the Spark applications.
>
>
> Kryo versions are very brittle.
>
> You'll
>
> -need to get an up to date/consistent version of Chill, which is where the
> transitive dependency on Kryo originates
> -rebuild spark depending on that chill release
>
> if you want hive integration, probably also rebuild Hive to be consistent
> too; the main reason Spark has its own Hive version is that
> Kryo version sharing.
>
> https://github.com/JoshRosen/hive/commits/release-1.2.1-spark2
>
> Kryo has repackaged their class locations between versions. This lets the
> versions co-exist, but probably also explains why your apps aren't picking
> up the diffs.
>
> Finally, keep an eye on this github PR
>
> https://github.com/twitter/chill/issues/252
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message