giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chui-Hui Chiu <cch...@tigers.lsu.edu>
Subject Re: Failed to compile Giraph for Hadoop YARN
Date Tue, 16 Jul 2013 16:23:42 GMT
Hello,

The "clean" argument solved my current problem.  However, I see more when I
go on.

On "http://giraph.apache.org/build.html",
(1) There're commands for secure and non-secure Hadoop.  What are the
commands for YARN?
(2) What do the compile, package, and test do respectively?  Do I have to
run each of them?  Do I have to run them in order?
When I ran "clean package", I saw the following error message.
=====
2013-07-16 11:07:42,383 FATAL [Thread-109] graph.GraphMapper
(GraphMapper.java:uncaughtException(112)) - uncaughtException:
OverrideExceptionHandler on thread Thread-109, msg = Failed to Start
ResourceManager, exiting...
org.apache.hadoop.yarn.YarnException: Failed to Start ResourceManager
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:567)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper$2.run(MiniYARNCluster.java:164)
Caused by: org.apache.hadoop.yarn.YarnException: java.net.BindException:
Problem binding to [0.0.0.0:8033] java.net.BindException: Address already
in use; For more details see:  http://wiki.apache.org/hadoop/BindException
at
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
at
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:63)
at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:52)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.start(AdminService.java:109)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
... 2 more
Caused by: java.net.BindException: Problem binding to [0.0.0.0:8033]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:716)
at org.apache.hadoop.ipc.Server.bind(Server.java:370)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:468)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1886)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:982)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:376)
at
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:351)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:705)
at
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
at
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
... 6 more
=====
Does the "package" command start another YARN cluster?


On "http://giraph.apache.org/pagerank.html",
(1) The last step, "Building and compiling page rank", is missing.
(2) I successfully compiled the giraph-examples and have a working Hadoop
YARN, how do I run the Page Rank program?

Many thanks,
Chui-hui


On Mon, Jul 15, 2013 at 12:53 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:

> I think something else is wrong then, this same build works fine for me. I
> think you need to do mvn targets "clean install" the first time, and
> "compile" from then on?
>
>
> On Mon, Jul 8, 2013 at 7:45 AM, Chui-Hui Chiu <cchiu1@tigers.lsu.edu>wrote:
>
>> I run
>>
>> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn compile
>>
>> on the source code of the Giraph-1.1.0-SNAPSHOT, which is released on
>> 05/21/2013, but still see the following message.
>>
>> =====
>>
>> [INFO] Executed tasks
>> [INFO]
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Building Apache Giraph Examples 1.1.0-SNAPSHOT
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Apache Giraph Parent .............................. SUCCESS
>> [1.335s]
>> [INFO] Apache Giraph Core ................................ SUCCESS
>> [41.165s]
>> [INFO] Apache Giraph Examples ............................ FAILURE
>> [0.122s]
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 43.270s
>> [INFO] Finished at: Mon Jul 08 09:37:16 CDT 2013
>> [INFO] Final Memory: 43M/724M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal on project giraph-examples: Could not
>> resolve dependencies for project
>> org.apache.giraph:giraph-examples:jar:1.1.0-SNAPSHOT: Failure to find
>> org.apache.giraph:giraph-core:jar:tests:1.1.0-SNAPSHOT in
>> http://repo1.maven.org/maven2 was cached in the local repository,
>> resolution will not be reattempted until the update interval of central has
>> elapsed or updates are forced -> [Help 1]
>>
>> =====
>>
>> When I apply the GIRAPH-688-1.patch, the patch utility reports that the
>> patch has applied to the source code.  I just compile it.
>>
>>
>> On Thu, Jul 4, 2013 at 11:56 AM, Eli Reisman <apache.mailbox@gmail.com>wrote:
>>
>>> Yes, glad this was brought up, I'll post to the site wiki with my
>>> explicit apologies for not doing it sooner.
>>>
>>> I posted the most recent "correct" way to build pure-YARN Giraph against
>>> any hadoop version on GIRAPH-688 here:
>>>
>>> https://issues.apache.org/jira/browse/GIRAPH-688
>>>
>>> The trick is, you have to put the -D mvn option first, and set the
>>> "hadoop.version" Maven var. The versions as seen in "hadoop.version" are
>>> -not- the same as the version profiles (-P option) that we build against.
>>> So the correct format is:
>>>
>>> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn clean install
>>>
>>> if you want to just compile and not create our build products as in your
>>> example, replace "clean install" with "compile". -P profile should be
>>> "hadoop_yarn" for pure YARN builds, and "hadoop.version" appears as the
>>> part of the hadoop version you see on the Apache Hadoop download site,
>>> -after- the "hadoop-" part. So "hadoop-2.0.3-alpha" becomes for us
>>> "-Dhadoop.version=2.0.3-alpha"
>>>
>>> To build Giraph so that it can run on a YARN-enabled Hadoop cluster that
>>> -also- uses MapReduce v2 as the interface to the cluster (Giraph will still
>>> think its running on MapReduce, not YARN) you might do this:
>>>
>>> mvn -Phadoop_2.0.3 clean install
>>>
>>> If you want to see how we label build profiles vs. "hadoop.version" in
>>> Giraph builds, check our POM.xml files
>>>
>>> Thanks! Sorry for the confusion, this is a confusing thing since there
>>> are 2 ways to "run Giraph on YARN" as detailed above.
>>>
>>>
>>>
>>>
>>> On Wed, Jul 3, 2013 at 10:16 AM, Avery Ching <aching@apache.org> wrote:
>>>
>>>> Eli, any thoughts?
>>>>
>>>>
>>>> On 7/3/13 9:27 AM, Chui-Hui Chiu wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I tried to compile the Giraph-1.1.0-SNAPSHOT for hadoop_2.0.3 or
>>>>> hadoop_yarn but all failed.
>>>>>
>>>>> The error message while the compile command is "mvn -Phadoop_yarn
>>>>> compile" is
>>>>> =====
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>>>> [1.320s]
>>>>> [INFO] Apache Giraph Core ..............................**.. FAILURE
>>>>> [12.508s]
>>>>> [INFO] Apache Giraph Examples ............................ SKIPPED
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] Total time: 14.473s
>>>>> [INFO] Finished at: Wed Jul 03 11:05:44 CDT 2013
>>>>> [INFO] Final Memory: 14M/216M
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [ERROR] Failed to execute goal on project giraph-core: Could not
>>>>> resolve dependencies for project org.apache.giraph:giraph-core:**jar:1.1.0-SNAPSHOT:
>>>>> The following artifacts could not be resolved: org.apache.hadoop:hadoop-
>>>>> **common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>>>> org.apache.hadoop:hadoop-**mapreduce-client-common:jar:**
>>>>> SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>>>>> org.apache.hadoop:hadoop-**mapreduce-client-core:jar:SET_**
>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>>>>> org.apache.hadoop:hadoop-yarn-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>>>> org.apache.hadoop:hadoop-yarn-**server-common:jar:SET_HADOOP_**
>>>>> VERSION_USING_MVN_DASH_D_**OPTION, org.apache.hadoop:hadoop-yarn-**
>>>>> server-resourcemanager:jar:**SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>>>>> org.apache.hadoop:hadoop-yarn-**server-nodemanager:jar:SET_**
>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>>>>> org.apache.hadoop:hadoop-yarn-**server-tests:jar:tests:SET_**
>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION: Could not find artifact
>>>>> org.apache.hadoop:hadoop-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION
>>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>>>>> =====
>>>>>
>>>>>
>>>>> The error message while the compile command is "mvn -Phadoop_2.0.3
>>>>> compile" is
>>>>> =====
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>>>> [12.695s]
>>>>> [INFO] Apache Giraph Core ..............................**.. SUCCESS
>>>>> [2:10.916s]
>>>>> [INFO] Apache Giraph Examples ............................ FAILURE
>>>>> [2.286s]
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [INFO] Total time: 2:26.530s
>>>>> [INFO] Finished at: Wed Jul 03 11:03:25 CDT 2013
>>>>> [INFO] Final Memory: 34M/348M
>>>>> [INFO] ------------------------------**------------------------------*
>>>>> *------------
>>>>> [ERROR] Failed to execute goal on project giraph-examples: Could not
>>>>> resolve dependencies for project org.apache.giraph:giraph-**examples:jar:1.1.0-SNAPSHOT:
>>>>> Could not find artifact org.apache.giraph:giraph-core:**jar:tests:1.1.0-SNAPSHOT
>>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>>>>> =====
>>>>>
>>>>> Do I miss anything?
>>>>>
>>>>>
>>>>> I also noticed that my Maven 3 download many files from the maven2
>>>>> folder on a remote server with the following prompt.
>>>>>
>>>>> Downloading: http://repo1.maven.org/maven2/**org/apache/hadoop/.<http://repo1.maven.org/maven2/org/apache/hadoop/.>
>>>>> ..
>>>>>
>>>>> Is this a problem?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Chui-hui
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message