giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: Failed to compile Giraph for Hadoop YARN
Date Thu, 04 Jul 2013 16:56:25 GMT
Yes, glad this was brought up, I'll post to the site wiki with my explicit
apologies for not doing it sooner.

I posted the most recent "correct" way to build pure-YARN Giraph against
any hadoop version on GIRAPH-688 here:

https://issues.apache.org/jira/browse/GIRAPH-688

The trick is, you have to put the -D mvn option first, and set the
"hadoop.version" Maven var. The versions as seen in "hadoop.version" are
-not- the same as the version profiles (-P option) that we build against.
So the correct format is:

mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn clean install

if you want to just compile and not create our build products as in your
example, replace "clean install" with "compile". -P profile should be
"hadoop_yarn" for pure YARN builds, and "hadoop.version" appears as the
part of the hadoop version you see on the Apache Hadoop download site,
-after- the "hadoop-" part. So "hadoop-2.0.3-alpha" becomes for us
"-Dhadoop.version=2.0.3-alpha"

To build Giraph so that it can run on a YARN-enabled Hadoop cluster that
-also- uses MapReduce v2 as the interface to the cluster (Giraph will still
think its running on MapReduce, not YARN) you might do this:

mvn -Phadoop_2.0.3 clean install

If you want to see how we label build profiles vs. "hadoop.version" in
Giraph builds, check our POM.xml files

Thanks! Sorry for the confusion, this is a confusing thing since there are
2 ways to "run Giraph on YARN" as detailed above.




On Wed, Jul 3, 2013 at 10:16 AM, Avery Ching <aching@apache.org> wrote:

> Eli, any thoughts?
>
>
> On 7/3/13 9:27 AM, Chui-Hui Chiu wrote:
>
>> Hello,
>>
>> I tried to compile the Giraph-1.1.0-SNAPSHOT for hadoop_2.0.3 or
>> hadoop_yarn but all failed.
>>
>> The error message while the compile command is "mvn -Phadoop_yarn
>> compile" is
>> =====
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Apache Giraph Parent .............................. SUCCESS
>> [1.320s]
>> [INFO] Apache Giraph Core ..............................**.. FAILURE
>> [12.508s]
>> [INFO] Apache Giraph Examples ............................ SKIPPED
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] BUILD FAILURE
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] Total time: 14.473s
>> [INFO] Finished at: Wed Jul 03 11:05:44 CDT 2013
>> [INFO] Final Memory: 14M/216M
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [ERROR] Failed to execute goal on project giraph-core: Could not resolve
>> dependencies for project org.apache.giraph:giraph-core:**jar:1.1.0-SNAPSHOT:
>> The following artifacts could not be resolved: org.apache.hadoop:hadoop-*
>> *common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>> org.apache.hadoop:hadoop-**mapreduce-client-common:jar:**
>> SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION, org.apache.hadoop:hadoop-**
>> mapreduce-client-core:jar:SET_**HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>> org.apache.hadoop:hadoop-yarn-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>> org.apache.hadoop:hadoop-yarn-**server-common:jar:SET_HADOOP_**
>> VERSION_USING_MVN_DASH_D_**OPTION, org.apache.hadoop:hadoop-yarn-**
>> server-resourcemanager:jar:**SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>> org.apache.hadoop:hadoop-yarn-**server-nodemanager:jar:SET_**
>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION, org.apache.hadoop:hadoop-yarn-*
>> *server-tests:jar:tests:SET_**HADOOP_VERSION_USING_MVN_DASH_**D_OPTION:
>> Could not find artifact org.apache.hadoop:hadoop-**
>> common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION in central (
>> http://repo1.maven.org/maven2**) -> [Help 1]
>> =====
>>
>>
>> The error message while the compile command is "mvn -Phadoop_2.0.3
>> compile" is
>> =====
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Apache Giraph Parent .............................. SUCCESS
>> [12.695s]
>> [INFO] Apache Giraph Core ..............................**.. SUCCESS
>> [2:10.916s]
>> [INFO] Apache Giraph Examples ............................ FAILURE
>> [2.286s]
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] BUILD FAILURE
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [INFO] Total time: 2:26.530s
>> [INFO] Finished at: Wed Jul 03 11:03:25 CDT 2013
>> [INFO] Final Memory: 34M/348M
>> [INFO] ------------------------------**------------------------------**
>> ------------
>> [ERROR] Failed to execute goal on project giraph-examples: Could not
>> resolve dependencies for project org.apache.giraph:giraph-**examples:jar:1.1.0-SNAPSHOT:
>> Could not find artifact org.apache.giraph:giraph-core:**jar:tests:1.1.0-SNAPSHOT
>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>> =====
>>
>> Do I miss anything?
>>
>>
>> I also noticed that my Maven 3 download many files from the maven2 folder
>> on a remote server with the following prompt.
>>
>> Downloading: http://repo1.maven.org/maven2/**org/apache/hadoop/.<http://repo1.maven.org/maven2/org/apache/hadoop/.>
>> ..
>>
>> Is this a problem?
>>
>>
>> Thanks,
>> Chui-hui
>>
>
>

Mime
View raw message