spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: How to compile Spark with customized Hadoop?
Date Sat, 10 Oct 2015 12:04:06 GMT

During development, I'd recommend giving Hadoop a version ending with -SNAPSHOT, and building
spark with maven, as mvn knows to refresh the snapshot every day.

you can do this in hadoop with

mvn versions:set 2.7.0.stevel-SNAPSHOT

if you are working on hadoop branch-2 or trunk direct, they come with -SNAPSHOT anyway, but
unless you build hadoop every morning, you may find maven pulls in the latest nightly builds
from the apache snapshot repository, which will cause chaos and confusion. This is also why
you must never have maven build which spans midnight in your time zone.

On 9 Oct 2015, at 22:31, Matei Zaharia <<>>

You can publish your version of Hadoop to your Maven cache with mvn publish (just give it
a different version number, e.g. 2.7.0a) and then pass that as the Hadoop version to Spark's
build (see


On Oct 9, 2015, at 3:10 PM, Dogtail L <<>>

Hi all,

I have modified Hadoop source code, and I want to compile Spark with my modified Hadoop. Do
you know how to do that? Great thanks!

View raw message