giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Fonseca <alexandrejorgefons...@gmail.com>
Subject Re: Issue while executing SimpleShortestPathsComputation
Date Sun, 08 Dec 2013 04:53:11 GMT
Hello Yong,

I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.2.0. 
Here's what I did:

	git clone https://git-wip-us.apache.org/repos/asf/giraph.git
	cd giraph
	git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
	mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile

With that I got valid jars which allowed me to run the SimpleShortestPaths 
example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:

	mkdir test_example
	ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
	$HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
giraph/input/tiny_graph.txt -vof 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca 
giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar

###########
Notes:
* Wasn't able to do mvn package as it complained about not having the 
dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a matter 
of changing the main pom.xml to include those modules with the hadoop 2.2.0 
target or remove those dependencies from the giraph-dist if they no longer 
apply.
* You may get some memory limit exceeded problems for the ApplicationMaster. 
To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in yarn-
site.xml or apply my patch to prevent allocation of heap size equal to total 
memory size, described at: https://issues.apache.org/jira/browse/GIRAPH-812
* In my case, the application didn't exit even after outputting the result and
having recognized that all nodes voted to terminate. I've opened an issue
in JIRA regarding this so if you suffer from the same, you can apply that patch
too: https://issues.apache.org/jira/browse/GIRAPH-811

###########
Relevant JIRA issues:
* https://issues.apache.org/jira/browse/GIRAPH-798
* https://issues.apache.org/jira/browse/GIRAPH-804
* https://issues.apache.org/jira/browse/GIRAPH-811
* https://issues.apache.org/jira/browse/GIRAPH-812

Best regards,
Alexandre

On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> Hi,
> 
> I also got this error when tried to build Giraph on top of hadoop-2.2.0
> using similar commands. Can anyone help?
> 
> Thanks,
> Yong
> 
> 
> 2013/12/6 Rohit Bhat <rohitbhat.nitk@gmail.com>
> 
> > I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2
> > commands and got similar errors in both the cases:
> > 
> > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests"
> > 
> > 
> > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> > package"
> > 
> > 
> > [INFO] Checking dependency versions
> > 
> > [ERROR] Found a problem with the direct dependency
> > commons-httpclient:commons-httpclient of the current project
> > 
> >   Expected version is 3.0.1
> >   
> >   Resolved version is 3.0.1
> >   
> >   Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
> > 
> > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> > 
> >   Resolved version is 2.4
> > 
> >   Version 2.4 was expected by artifact:
> > commons-configuration:commons-configuration
> > 
> >   Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
> > 
> > [ERROR] Found a problem with the dependency
> > commons-logging:commons-logging
> > 
> >   Resolved version is 1.0.4
> >   
> >   Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
> > 
> > [ERROR] Found a problem with the dependency io.netty:netty
> > 
> >   Resolved version is 3.5.3.Final
> >   
> >   Version 3.6.2.Final was expected by artifacts: org.apache.
> > 
> > hadoop:hadoop-mapreduce-client-common, org.apache.
> > hadoop:hadoop-mapreduce-client-core
> > 
> > [ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
> > 
> >   Resolved version is 3.3.3
> >   
> >   Version 3.4.5 was expected by artifacts:
> >   org.apache.hadoop:hadoop-common,
> > 
> > org.apache.hadoop:hadoop-mapreduce-client-common
> > 
> > 
> > Am i missing something?
> > 
> > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar 
<ameya.vilankar@gmail.com>wrote:
> >> Yes. Use this:
> >> 
> >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> >> 
> >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang 
<manuellagang@gmail.com>wrote:
> >>> I get the same error when I compile Giraph against the default hadoop
> >>> version (0.20.203.0), but my project that uses Giraph uses a more
> >>> recent hadoop version. Did you set the hadoop version via a maven
> >>> profile
> >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
> >>> 
> >>> Presumably, there's a way to compile Giraph for Yarn. There's even a
> >>> Yarn profile in the pom.xml. I personally haven't had success building
> >>> it
> >>> though.
> >>> 
> >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat 
<rohitbhat.nitk@gmail.com>wrote:
> >>>> Hi,
> >>>> 
> >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> >>>> installed Giraph on top of it. I have been trying to
> >>>> execute SimpleShortestPathsComputation example but i am getting the
> >>>> error:
> >>>> 
> >>>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
> >>>> When using LocalJobRunner, you cannot run in split master / worker mode
> >>>> since there is only 1 task at a time!"
> >>>> 
> >>>> I fixed this by adding an entry in mapred-site.xml for
> >>>> mapreduce.jobtracker.address but now i am getting a different error:
> >>>> 
> >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
> >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
> >>>> expected*
> >>>> 
> >>>>  at
> >>>> 
> >>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> >>>> java:43) at
> >>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> >>>> 56)
> >>>> 
> >>>>  at
> >>>> 
> >>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >>>> 
> >>>>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >>>> 
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> 
> >>>>  at javax.security.auth.Subject.doAs(Subject.java:415)
> >>>> 
> >>>> at
> >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>>> on.java:1491)>>>> 
> >>>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >>>> 
> >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> >>>> 
> >>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> >>>> 
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>>> 
> >>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >>>> 
> >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> >>>> 
> >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> 
> >>>> at
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>>> va:57)>>>> 
> >>>>  at
> >>>> 
> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> 
> >>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >>>> 
> >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> >>>> 
> >>>> Thanks,
> >>>> Rohit

Mime
View raw message