spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Francke <lars.fran...@gmail.com>
Subject Re: YARN Maven build questions
Date Fri, 28 Feb 2014 17:41:00 GMT
Hey,

so currently it doesn't work because of
<https://github.com/apache/spark/pull/6#issuecomment-36343187>

IntelliJ reports a lot of warnings with default settings and I haven't
found a way to tell IntellJ to use different Hadoop versions yet.
mvn clean compile -Pyarn fails as well (compilation errror

Your command works indeed. Default yarn version is 0.23.7 which
doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)

I was basically trying to follow the documentation:
<http://spark.incubator.apache.org/docs/latest/building-with-maven.html>

mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
-Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
-Pyarn-alpha

Thanks for showing me a configuration that works. Unfortunately the
default ones and at least one of the documented ones fail.

Cheers,
Lars


On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tgraves_cs@yahoo.com> wrote:
> what build command are you using?    What do you mean when you say YARN branch?
>
> The yarn builds have been working fine for me with maven.   Build command I use against
hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package
-DskipTests
>
> Tom
>
>
>
> On Friday, February 28, 2014 6:14 AM, Lars Francke <lars.francke@gmail.com> wrote:
>
> Hey,
>
> I'm trying to dig into Spark's code but am running into a couple of problems.
>
> 1) The yarn-common directory is not included in the Maven build
> causing things to fail because the dependency is missing. If I see the
> history correct it used to be a Maven module but is not anymore.
>
> 2) When I try to include the yarn-common directory in the build things
> start going bad. Compilation failures all over the place and I think
> there are some dependency issues in there as well.
>
> This leads me to believe that either the Maven build system isn't
> maintained for YARN or the whole YARN branch isn't. What's the status
> here?
>
> Without YARN things build fine for me using Maven.
>
> Thanks for your help.
>
> Cheers,
> Lars

Mime
View raw message