giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: Issue while executing SimpleShortestPathsComputation
Date Sun, 08 Dec 2013 18:02:28 GMT
Hey all,

So. I think from more recent emails Rohit got this figured out, but, a
couple points just so they out there again:

Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop 2.0.03-alpha
or (with a patch thats still floating around on GIRAPH-687 ticket) against
Hadoop 2.0.4 and 2.0.5 alpha.

Much better way to go is to build against Hadoop 2.2.x line using Giraph
1.1.x's YARN profile. This features a stable API, some basic support for
security, and a concurrency fix that could bite you when launching many
workers.

Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN profile.
This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
but riding on Mapper task instead of running the workers in YARN containers.



On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
alexandrejorgefonseca@gmail.com> wrote:

> Hello Yong,
>
> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.2.0.
> Here's what I did:
>
>         git clone https://git-wip-us.apache.org/repos/asf/giraph.git
>         cd giraph
>         git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
>         mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>
> With that I got valid jars which allowed me to run the SimpleShortestPaths
> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>
>         mkdir test_example
>         ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>         $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip
> giraph/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> hadoop-2.2.0-jar-with-dependencies.jar
>
> ###########
> Notes:
> * Wasn't able to do mvn package as it complained about not having the
> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
> matter
> of changing the main pom.xml to include those modules with the hadoop 2.2.0
> target or remove those dependencies from the giraph-dist if they no longer
> apply.
> * You may get some memory limit exceeded problems for the
> ApplicationMaster.
> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
> yarn-
> site.xml or apply my patch to prevent allocation of heap size equal to
> total
> memory size, described at:
> https://issues.apache.org/jira/browse/GIRAPH-812
> * In my case, the application didn't exit even after outputting the result
> and
> having recognized that all nodes voted to terminate. I've opened an issue
> in JIRA regarding this so if you suffer from the same, you can apply that
> patch
> too: https://issues.apache.org/jira/browse/GIRAPH-811
>
> ###########
> Relevant JIRA issues:
> * https://issues.apache.org/jira/browse/GIRAPH-798
> * https://issues.apache.org/jira/browse/GIRAPH-804
> * https://issues.apache.org/jira/browse/GIRAPH-811
> * https://issues.apache.org/jira/browse/GIRAPH-812
>
> Best regards,
> Alexandre
>
> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> > Hi,
> >
> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
> > using similar commands. Can anyone help?
> >
> > Thanks,
> > Yong
> >
> >
> > 2013/12/6 Rohit Bhat <rohitbhat.nitk@gmail.com>
> >
> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> following 2
> > > commands and got similar errors in both the cases:
> > >
> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> -DskipTests"
> > >
> > >
> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> > > package"
> > >
> > >
> > > [INFO] Checking dependency versions
> > >
> > > [ERROR] Found a problem with the direct dependency
> > > commons-httpclient:commons-httpclient of the current project
> > >
> > >   Expected version is 3.0.1
> > >
> > >   Resolved version is 3.0.1
> > >
> > >   Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> > >
> > >   Resolved version is 2.4
> > >
> > >   Version 2.4 was expected by artifact:
> > > commons-configuration:commons-configuration
> > >
> > >   Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency
> > > commons-logging:commons-logging
> > >
> > >   Resolved version is 1.0.4
> > >
> > >   Version 1.1.1 was expected by artifact:
> org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency io.netty:netty
> > >
> > >   Resolved version is 3.5.3.Final
> > >
> > >   Version 3.6.2.Final was expected by artifacts: org.apache.
> > >
> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> > > hadoop:hadoop-mapreduce-client-core
> > >
> > > [ERROR] Found a problem with the dependency
> org.apache.zookeeper:zookeeper
> > >
> > >   Resolved version is 3.3.3
> > >
> > >   Version 3.4.5 was expected by artifacts:
> > >   org.apache.hadoop:hadoop-common,
> > >
> > > org.apache.hadoop:hadoop-mapreduce-client-common
> > >
> > >
> > > Am i missing something?
> > >
> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> <ameya.vilankar@gmail.com>wrote:
> > >> Yes. Use this:
> > >>
> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> > >>
> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> <manuellagang@gmail.com>wrote:
> > >>> I get the same error when I compile Giraph against the default hadoop
> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
> > >>> recent hadoop version. Did you set the hadoop version via a maven
> > >>> profile
> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
> > >>>
> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even
a
> > >>> Yarn profile in the pom.xml. I personally haven't had success
> building
> > >>> it
> > >>> though.
> > >>>
> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> <rohitbhat.nitk@gmail.com>wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> > >>>> installed Giraph on top of it. I have been trying to
> > >>>> execute SimpleShortestPathsComputation example but i am getting
the
> > >>>> error:
> > >>>>
> > >>>> "java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration:
> > >>>> When using LocalJobRunner, you cannot run in split master / worker
> mode
> > >>>> since there is only 1 task at a time!"
> > >>>>
> > >>>> I fixed this by adding an entry in mapred-site.xml for
> > >>>> mapreduce.jobtracker.address but now i am getting a different error:
> > >>>>
> > >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class
> was
> > >>>> expected*
> > >>>>
> > >>>>  at
> > >>>>
> > >>>>
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> > >>>> java:43) at
> > >>>>
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> > >>>> 56)
> > >>>>
> > >>>>  at
> > >>>>
> > >>>>
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> > >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> > >>>>
> > >>>>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> > >>>>
> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> > >>>>
> > >>>>  at javax.security.auth.Subject.doAs(Subject.java:415)
> > >>>>
> > >>>> at
> > >>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> > >>>> on.java:1491)>>>>
> > >>>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> > >>>>
> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> > >>>>
> > >>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> > >>>>
> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >>>>
> > >>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >>>>
> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> > >>>>
> > >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >>>>
> > >>>> at
> > >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> > >>>> va:57)>>>>
> > >>>>  at
> > >>>>
> > >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> > >>>>
> > >>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > >>>>
> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> > >>>>
> > >>>> Thanks,
> > >>>> Rohit
>

Mime
View raw message