spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Francke <>
Subject Re: YARN Maven build questions
Date Tue, 04 Mar 2014 08:50:00 GMT

sorry to bother again.

As a newbie to the project it's hard to judge whether I'm doing
anything wrong, the documentation is outdated or the Maven/SBT files
have diverged from the actual code by defining older/now incompatible
versions or something else going wrong.

Any hints?

Also an unrelated note/question: I see tons of pull requests being
accepted without a JIRA but the documentation says to create a JIRA
issue first[1]. So I assume it's okay to just send pull requests?

Thanks for your help.


[1] <>

On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <> wrote:
> Hey,
> so currently it doesn't work because of
> <>
> IntelliJ reports a lot of warnings with default settings and I haven't
> found a way to tell IntellJ to use different Hadoop versions yet.
> mvn clean compile -Pyarn fails as well (compilation errror
> Your command works indeed. Default yarn version is 0.23.7 which
> doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
> I was basically trying to follow the documentation:
> <>
> mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
> -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
> -Pyarn-alpha
> Thanks for showing me a configuration that works. Unfortunately the
> default ones and at least one of the documented ones fail.
> Cheers,
> Lars
> On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <> wrote:
>> what build command are you using?    What do you mean when you say YARN branch?
>> The yarn builds have been working fine for me with maven.   Build command I use against
hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package
>> Tom
>> On Friday, February 28, 2014 6:14 AM, Lars Francke <>
>> Hey,
>> I'm trying to dig into Spark's code but am running into a couple of problems.
>> 1) The yarn-common directory is not included in the Maven build
>> causing things to fail because the dependency is missing. If I see the
>> history correct it used to be a Maven module but is not anymore.
>> 2) When I try to include the yarn-common directory in the build things
>> start going bad. Compilation failures all over the place and I think
>> there are some dependency issues in there as well.
>> This leads me to believe that either the Maven build system isn't
>> maintained for YARN or the whole YARN branch isn't. What's the status
>> here?
>> Without YARN things build fine for me using Maven.
>> Thanks for your help.
>> Cheers,
>> Lars

View raw message