giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: Failed to compile Giraph for Hadoop YARN
Date Thu, 04 Jul 2013 16:59:09 GMT
One last note: all of this versioning is subject to which Hadoop-2.0.x line
versions are hosted where we get our Maven dependencies. So as 2.0.4 and
2.0.5 have arrived, it won't be long before some of those mirror sites will
stop hosting 2.0.3 -- be ready. This happened to me with 2.0.1 and 2.0.2
while I was writing the port to YARN!



On Thu, Jul 4, 2013 at 9:56 AM, Eli Reisman <apache.mailbox@gmail.com>wrote:

> Yes, glad this was brought up, I'll post to the site wiki with my explicit
> apologies for not doing it sooner.
>
> I posted the most recent "correct" way to build pure-YARN Giraph against
> any hadoop version on GIRAPH-688 here:
>
> https://issues.apache.org/jira/browse/GIRAPH-688
>
> The trick is, you have to put the -D mvn option first, and set the
> "hadoop.version" Maven var. The versions as seen in "hadoop.version" are
> -not- the same as the version profiles (-P option) that we build against.
> So the correct format is:
>
> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn clean install
>
> if you want to just compile and not create our build products as in your
> example, replace "clean install" with "compile". -P profile should be
> "hadoop_yarn" for pure YARN builds, and "hadoop.version" appears as the
> part of the hadoop version you see on the Apache Hadoop download site,
> -after- the "hadoop-" part. So "hadoop-2.0.3-alpha" becomes for us
> "-Dhadoop.version=2.0.3-alpha"
>
> To build Giraph so that it can run on a YARN-enabled Hadoop cluster that
> -also- uses MapReduce v2 as the interface to the cluster (Giraph will still
> think its running on MapReduce, not YARN) you might do this:
>
> mvn -Phadoop_2.0.3 clean install
>
> If you want to see how we label build profiles vs. "hadoop.version" in
> Giraph builds, check our POM.xml files
>
> Thanks! Sorry for the confusion, this is a confusing thing since there are
> 2 ways to "run Giraph on YARN" as detailed above.
>
>
>
>
> On Wed, Jul 3, 2013 at 10:16 AM, Avery Ching <aching@apache.org> wrote:
>
>> Eli, any thoughts?
>>
>>
>> On 7/3/13 9:27 AM, Chui-Hui Chiu wrote:
>>
>>> Hello,
>>>
>>> I tried to compile the Giraph-1.1.0-SNAPSHOT for hadoop_2.0.3 or
>>> hadoop_yarn but all failed.
>>>
>>> The error message while the compile command is "mvn -Phadoop_yarn
>>> compile" is
>>> =====
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>> [1.320s]
>>> [INFO] Apache Giraph Core ..............................**.. FAILURE
>>> [12.508s]
>>> [INFO] Apache Giraph Examples ............................ SKIPPED
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] BUILD FAILURE
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] Total time: 14.473s
>>> [INFO] Finished at: Wed Jul 03 11:05:44 CDT 2013
>>> [INFO] Final Memory: 14M/216M
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [ERROR] Failed to execute goal on project giraph-core: Could not resolve
>>> dependencies for project org.apache.giraph:giraph-core:**jar:1.1.0-SNAPSHOT:
>>> The following artifacts could not be resolved: org.apache.hadoop:hadoop-
>>> **common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>> org.apache.hadoop:hadoop-**mapreduce-client-common:jar:**
>>> SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION, org.apache.hadoop:hadoop-*
>>> *mapreduce-client-core:jar:SET_**HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>>> org.apache.hadoop:hadoop-yarn-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>> org.apache.hadoop:hadoop-yarn-**server-common:jar:SET_HADOOP_**
>>> VERSION_USING_MVN_DASH_D_**OPTION, org.apache.hadoop:hadoop-yarn-**
>>> server-resourcemanager:jar:**SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>>> org.apache.hadoop:hadoop-yarn-**server-nodemanager:jar:SET_**
>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION, org.apache.hadoop:hadoop-yarn-
>>> **server-tests:jar:tests:SET_**HADOOP_VERSION_USING_MVN_DASH_**D_OPTION:
>>> Could not find artifact org.apache.hadoop:hadoop-**
>>> common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION in central (
>>> http://repo1.maven.org/maven2**) -> [Help 1]
>>> =====
>>>
>>>
>>> The error message while the compile command is "mvn -Phadoop_2.0.3
>>> compile" is
>>> =====
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>> [12.695s]
>>> [INFO] Apache Giraph Core ..............................**.. SUCCESS
>>> [2:10.916s]
>>> [INFO] Apache Giraph Examples ............................ FAILURE
>>> [2.286s]
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] BUILD FAILURE
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [INFO] Total time: 2:26.530s
>>> [INFO] Finished at: Wed Jul 03 11:03:25 CDT 2013
>>> [INFO] Final Memory: 34M/348M
>>> [INFO] ------------------------------**------------------------------**
>>> ------------
>>> [ERROR] Failed to execute goal on project giraph-examples: Could not
>>> resolve dependencies for project org.apache.giraph:giraph-**examples:jar:1.1.0-SNAPSHOT:
>>> Could not find artifact org.apache.giraph:giraph-core:**jar:tests:1.1.0-SNAPSHOT
>>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>>> =====
>>>
>>> Do I miss anything?
>>>
>>>
>>> I also noticed that my Maven 3 download many files from the maven2
>>> folder on a remote server with the following prompt.
>>>
>>> Downloading: http://repo1.maven.org/maven2/**org/apache/hadoop/.<http://repo1.maven.org/maven2/org/apache/hadoop/.>
>>> ..
>>>
>>> Is this a problem?
>>>
>>>
>>> Thanks,
>>> Chui-hui
>>>
>>
>>
>

Mime
View raw message