Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <3181C752-A22E-4F52-AA89-6A55F1092F58@hortonworks.com>
References: <CAF6hzxwZ8i9sYgZz9mAXPi-TszVMKZ=ajO3xSCPXvz2uh0AuHQ@mail.gmail.com>
 <3181C752-A22E-4F52-AA89-6A55F1092F58@hortonworks.com>
From: Prasun Ratn <prasun.ratn@gmail.com>
Date: Mon, 17 Oct 2016 19:29:26 +0530
Message-ID: <CAF6hzxxSX2E72WXeb1xpZ8WsLE07gb3utu3mvYTyZbf_4=SfMQ@mail.gmail.com>
Subject: Re: trying to use Spark applications with modified Kryo
To: Steve Loughran <stevel@hortonworks.com>
Cc: Apache Spark Dev <dev@spark.apache.org>
Content-Type: text/plain; charset=UTF-8
archived-at: Mon, 17 Oct 2016 14:00:02 -0000

Thanks a lot Steve!

On Mon, Oct 17, 2016 at 4:59 PM, Steve Loughran <stevel@hortonworks.com> wrote:
>
> On 17 Oct 2016, at 10:02, Prasun Ratn <prasun.ratn@gmail.com> wrote:
>
> Hi
>
> I want to run some Spark applications with some changes in Kryo serializer.
>
> Please correct me, but I think I need to recompile spark (instead of
> just the Spark applications) in order to use the newly built Kryo
> serializer?
>
> I obtained Kryo 3.0.3 source and built it (mvn package install).
>
> Next, I took the source code for Spark 2.0.1 and built it (build/mvn
> -X -DskipTests -Dhadoop.version=2.6.0 clean package)
>
> I then compiled the Spark applications.
>
> However, I am not seeing my Kryo changes when I run the Spark applications.
>
>
> Kryo versions are very brittle.
>
> You'll
>
> -need to get an up to date/consistent version of Chill, which is where the
> transitive dependency on Kryo originates
> -rebuild spark depending on that chill release
>
> if you want hive integration, probably also rebuild Hive to be consistent
> too; the main reason Spark has its own Hive version is that
> Kryo version sharing.
>
> https://github.com/JoshRosen/hive/commits/release-1.2.1-spark2
>
> Kryo has repackaged their class locations between versions. This lets the
> versions co-exist, but probably also explains why your apps aren't picking
> up the diffs.
>
> Finally, keep an eye on this github PR
>
> https://github.com/twitter/chill/issues/252
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org