giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: Failed to compile Giraph for Hadoop YARN
Date Fri, 26 Jul 2013 21:00:31 GMT
Hopefully my other email helped, but:

1) no secure version for the YARN port yet
2) you shoudl just run the mvn install or mvn verify tasks, the rest come
with. for detailed command lines to do this on your version of Hadoop/YARN,
see the other email.

Thanks!



On Tue, Jul 16, 2013 at 9:23 AM, Chui-Hui Chiu <cchiu1@tigers.lsu.edu>wrote:

> Hello,
>
> The "clean" argument solved my current problem.  However, I see more when
> I go on.
>
> On "http://giraph.apache.org/build.html",
> (1) There're commands for secure and non-secure Hadoop.  What are the
> commands for YARN?
> (2) What do the compile, package, and test do respectively?  Do I have to
> run each of them?  Do I have to run them in order?
> When I ran "clean package", I saw the following error message.
> =====
> 2013-07-16 11:07:42,383 FATAL [Thread-109] graph.GraphMapper
> (GraphMapper.java:uncaughtException(112)) - uncaughtException:
> OverrideExceptionHandler on thread Thread-109, msg = Failed to Start
> ResourceManager, exiting...
> org.apache.hadoop.yarn.YarnException: Failed to Start ResourceManager
> at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
>  at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:567)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper$2.run(MiniYARNCluster.java:164)
> Caused by: org.apache.hadoop.yarn.YarnException: java.net.BindException:
> Problem binding to [0.0.0.0:8033] java.net.BindException: Address already
> in use; For more details see:  http://wiki.apache.org/hadoop/BindException
>  at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:63)
>  at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.start(AdminService.java:109)
>  at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> ... 2 more
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:8033]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
>  at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:716)
> at org.apache.hadoop.ipc.Server.bind(Server.java:370)
>  at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:468)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:1886)
>  at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:982)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:376)
>  at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:351)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:705)
>  at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
>  ... 6 more
> =====
> Does the "package" command start another YARN cluster?
>
>
> On "http://giraph.apache.org/pagerank.html",
> (1) The last step, "Building and compiling page rank", is missing.
> (2) I successfully compiled the giraph-examples and have a working Hadoop
> YARN, how do I run the Page Rank program?
>
> Many thanks,
> Chui-hui
>
>
> On Mon, Jul 15, 2013 at 12:53 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:
>
>> I think something else is wrong then, this same build works fine for me.
>> I think you need to do mvn targets "clean install" the first time, and
>> "compile" from then on?
>>
>>
>> On Mon, Jul 8, 2013 at 7:45 AM, Chui-Hui Chiu <cchiu1@tigers.lsu.edu>wrote:
>>
>>> I run
>>>
>>> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn compile
>>>
>>> on the source code of the Giraph-1.1.0-SNAPSHOT, which is released on
>>> 05/21/2013, but still see the following message.
>>>
>>> =====
>>>
>>> [INFO] Executed tasks
>>> [INFO]
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Building Apache Giraph Examples 1.1.0-SNAPSHOT
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>> [1.335s]
>>> [INFO] Apache Giraph Core ................................ SUCCESS
>>> [41.165s]
>>> [INFO] Apache Giraph Examples ............................ FAILURE
>>> [0.122s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 43.270s
>>> [INFO] Finished at: Mon Jul 08 09:37:16 CDT 2013
>>> [INFO] Final Memory: 43M/724M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal on project giraph-examples: Could not
>>> resolve dependencies for project
>>> org.apache.giraph:giraph-examples:jar:1.1.0-SNAPSHOT: Failure to find
>>> org.apache.giraph:giraph-core:jar:tests:1.1.0-SNAPSHOT in
>>> http://repo1.maven.org/maven2 was cached in the local repository,
>>> resolution will not be reattempted until the update interval of central has
>>> elapsed or updates are forced -> [Help 1]
>>>
>>> =====
>>>
>>> When I apply the GIRAPH-688-1.patch, the patch utility reports that the
>>> patch has applied to the source code.  I just compile it.
>>>
>>>
>>> On Thu, Jul 4, 2013 at 11:56 AM, Eli Reisman <apache.mailbox@gmail.com>wrote:
>>>
>>>> Yes, glad this was brought up, I'll post to the site wiki with my
>>>> explicit apologies for not doing it sooner.
>>>>
>>>> I posted the most recent "correct" way to build pure-YARN Giraph
>>>> against any hadoop version on GIRAPH-688 here:
>>>>
>>>> https://issues.apache.org/jira/browse/GIRAPH-688
>>>>
>>>> The trick is, you have to put the -D mvn option first, and set the
>>>> "hadoop.version" Maven var. The versions as seen in "hadoop.version" are
>>>> -not- the same as the version profiles (-P option) that we build against.
>>>> So the correct format is:
>>>>
>>>> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn clean install
>>>>
>>>> if you want to just compile and not create our build products as in
>>>> your example, replace "clean install" with "compile". -P profile should be
>>>> "hadoop_yarn" for pure YARN builds, and "hadoop.version" appears as the
>>>> part of the hadoop version you see on the Apache Hadoop download site,
>>>> -after- the "hadoop-" part. So "hadoop-2.0.3-alpha" becomes for us
>>>> "-Dhadoop.version=2.0.3-alpha"
>>>>
>>>> To build Giraph so that it can run on a YARN-enabled Hadoop cluster
>>>> that -also- uses MapReduce v2 as the interface to the cluster (Giraph will
>>>> still think its running on MapReduce, not YARN) you might do this:
>>>>
>>>> mvn -Phadoop_2.0.3 clean install
>>>>
>>>> If you want to see how we label build profiles vs. "hadoop.version" in
>>>> Giraph builds, check our POM.xml files
>>>>
>>>> Thanks! Sorry for the confusion, this is a confusing thing since there
>>>> are 2 ways to "run Giraph on YARN" as detailed above.
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jul 3, 2013 at 10:16 AM, Avery Ching <aching@apache.org> wrote:
>>>>
>>>>> Eli, any thoughts?
>>>>>
>>>>>
>>>>> On 7/3/13 9:27 AM, Chui-Hui Chiu wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I tried to compile the Giraph-1.1.0-SNAPSHOT for hadoop_2.0.3 or
>>>>>> hadoop_yarn but all failed.
>>>>>>
>>>>>> The error message while the compile command is "mvn -Phadoop_yarn
>>>>>> compile" is
>>>>>> =====
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>>>>> [1.320s]
>>>>>> [INFO] Apache Giraph Core ..............................**.. FAILURE
>>>>>> [12.508s]
>>>>>> [INFO] Apache Giraph Examples ............................ SKIPPED
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] Total time: 14.473s
>>>>>> [INFO] Finished at: Wed Jul 03 11:05:44 CDT 2013
>>>>>> [INFO] Final Memory: 14M/216M
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [ERROR] Failed to execute goal on project giraph-core: Could not
>>>>>> resolve dependencies for project org.apache.giraph:giraph-core:**jar:1.1.0-SNAPSHOT:
>>>>>> The following artifacts could not be resolved: org.apache.hadoop:hadoop-
>>>>>> **common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>>>>> org.apache.hadoop:hadoop-**mapreduce-client-common:jar:**
>>>>>> SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>>>>>> org.apache.hadoop:hadoop-**mapreduce-client-core:jar:SET_**
>>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>>>>>> org.apache.hadoop:hadoop-yarn-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION,
>>>>>> org.apache.hadoop:hadoop-yarn-**server-common:jar:SET_HADOOP_**
>>>>>> VERSION_USING_MVN_DASH_D_**OPTION, org.apache.hadoop:hadoop-yarn-**
>>>>>> server-resourcemanager:jar:**SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION,
>>>>>> org.apache.hadoop:hadoop-yarn-**server-nodemanager:jar:SET_**
>>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION,
>>>>>> org.apache.hadoop:hadoop-yarn-**server-tests:jar:tests:SET_**
>>>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION: Could not find artifact
>>>>>> org.apache.hadoop:hadoop-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION
>>>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>>>>>> =====
>>>>>>
>>>>>>
>>>>>> The error message while the compile command is "mvn -Phadoop_2.0.3
>>>>>> compile" is
>>>>>> =====
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>>>>> [12.695s]
>>>>>> [INFO] Apache Giraph Core ..............................**.. SUCCESS
>>>>>> [2:10.916s]
>>>>>> [INFO] Apache Giraph Examples ............................ FAILURE
>>>>>> [2.286s]
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [INFO] Total time: 2:26.530s
>>>>>> [INFO] Finished at: Wed Jul 03 11:03:25 CDT 2013
>>>>>> [INFO] Final Memory: 34M/348M
>>>>>> [INFO] ------------------------------**------------------------------
>>>>>> **------------
>>>>>> [ERROR] Failed to execute goal on project giraph-examples: Could
not
>>>>>> resolve dependencies for project org.apache.giraph:giraph-**examples:jar:1.1.0-SNAPSHOT:
>>>>>> Could not find artifact org.apache.giraph:giraph-core:**jar:tests:1.1.0-SNAPSHOT
>>>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1]
>>>>>> =====
>>>>>>
>>>>>> Do I miss anything?
>>>>>>
>>>>>>
>>>>>> I also noticed that my Maven 3 download many files from the maven2
>>>>>> folder on a remote server with the following prompt.
>>>>>>
>>>>>> Downloading: http://repo1.maven.org/maven2/**org/apache/hadoop/.<http://repo1.maven.org/maven2/org/apache/hadoop/.>
>>>>>> ..
>>>>>>
>>>>>> Is this a problem?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Chui-hui
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message