spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Francke <lars.fran...@gmail.com>
Subject Re: YARN Maven build questions
Date Tue, 04 Mar 2014 08:50:00 GMT
Hi,

sorry to bother again.

As a newbie to the project it's hard to judge whether I'm doing
anything wrong, the documentation is outdated or the Maven/SBT files
have diverged from the actual code by defining older/now incompatible
versions or something else going wrong.

Any hints?

Also an unrelated note/question: I see tons of pull requests being
accepted without a JIRA but the documentation says to create a JIRA
issue first[1]. So I assume it's okay to just send pull requests?

Thanks for your help.

Cheers,
Lars

[1] <https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark>

On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <lars.francke@gmail.com> wrote:
> Hey,
>
> so currently it doesn't work because of
> <https://github.com/apache/spark/pull/6#issuecomment-36343187>
>
> IntelliJ reports a lot of warnings with default settings and I haven't
> found a way to tell IntellJ to use different Hadoop versions yet.
> mvn clean compile -Pyarn fails as well (compilation errror
>
> Your command works indeed. Default yarn version is 0.23.7 which
> doesn't seem to work with the default 2.2.0 Hadoop version (anymore?)
>
> I was basically trying to follow the documentation:
> <http://spark.incubator.apache.org/docs/latest/building-with-maven.html>
>
> mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0
> -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile
> -Pyarn-alpha
>
> Thanks for showing me a configuration that works. Unfortunately the
> default ones and at least one of the documented ones fail.
>
> Cheers,
> Lars
>
>
> On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tgraves_cs@yahoo.com> wrote:
>> what build command are you using?    What do you mean when you say YARN branch?
>>
>> The yarn builds have been working fine for me with maven.   Build command I use against
hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn clean package
-DskipTests
>>
>> Tom
>>
>>
>>
>> On Friday, February 28, 2014 6:14 AM, Lars Francke <lars.francke@gmail.com>
wrote:
>>
>> Hey,
>>
>> I'm trying to dig into Spark's code but am running into a couple of problems.
>>
>> 1) The yarn-common directory is not included in the Maven build
>> causing things to fail because the dependency is missing. If I see the
>> history correct it used to be a Maven module but is not anymore.
>>
>> 2) When I try to include the yarn-common directory in the build things
>> start going bad. Compilation failures all over the place and I think
>> there are some dependency issues in there as well.
>>
>> This leads me to believe that either the Maven build system isn't
>> maintained for YARN or the whole YARN branch isn't. What's the status
>> here?
>>
>> Without YARN things build fine for me using Maven.
>>
>> Thanks for your help.
>>
>> Cheers,
>> Lars

Mime
View raw message