giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohit Bhat <rohitbhat.n...@gmail.com>
Subject Re: Issue while executing SimpleShortestPathsComputation
Date Mon, 09 Dec 2013 15:50:35 GMT
Hi Eli/Alexandre,

I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using the
steps mentioned by Alexandre, but again i am getting the same error while
running the example:

Command:

$HADOOP_HOME/bin/hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /giraphInput/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
giraph.zkList=localhost:2181 -yj
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar

ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1386601013698_0005 failed
2 times due to AM Container for appattempt_1386601013698_0005_000002 exited
with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)


.Failing this attempt.. Failing the application.

I checked the Hadoop logs as well but couldn't find anything meaningful
there too.

Thanks,
Rohit


On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:

> Hey all,
>
> So. I think from more recent emails Rohit got this figured out, but, a
> couple points just so they out there again:
>
> Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687
> ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
>
> Much better way to go is to build against Hadoop 2.2.x line using Giraph
> 1.1.x's YARN profile. This features a stable API, some basic support for
> security, and a concurrency fix that could bite you when launching many
> workers.
>
> Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
> build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN profile.
> This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
> but riding on Mapper task instead of running the workers in YARN containers.
>
>
>
> On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> alexandrejorgefonseca@gmail.com> wrote:
>
>> Hello Yong,
>>
>> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
>> hadoop-2.2.0.
>> Here's what I did:
>>
>>         git clone https://git-wip-us.apache.org/repos/asf/giraph.git
>>         cd giraph
>>         git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
>>         mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>>
>> With that I got valid jars which allowed me to run the SimpleShortestPaths
>> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>>
>>         mkdir test_example
>>         ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
>> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>         $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip
>> giraph/input/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
>> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar
>>
>> ###########
>> Notes:
>> * Wasn't able to do mvn package as it complained about not having the
>> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
>> matter
>> of changing the main pom.xml to include those modules with the hadoop
>> 2.2.0
>> target or remove those dependencies from the giraph-dist if they no longer
>> apply.
>> * You may get some memory limit exceeded problems for the
>> ApplicationMaster.
>> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
>> yarn-
>> site.xml or apply my patch to prevent allocation of heap size equal to
>> total
>> memory size, described at:
>> https://issues.apache.org/jira/browse/GIRAPH-812
>> * In my case, the application didn't exit even after outputting the
>> result and
>> having recognized that all nodes voted to terminate. I've opened an issue
>> in JIRA regarding this so if you suffer from the same, you can apply that
>> patch
>> too: https://issues.apache.org/jira/browse/GIRAPH-811
>>
>> ###########
>> Relevant JIRA issues:
>> * https://issues.apache.org/jira/browse/GIRAPH-798
>> * https://issues.apache.org/jira/browse/GIRAPH-804
>> * https://issues.apache.org/jira/browse/GIRAPH-811
>> * https://issues.apache.org/jira/browse/GIRAPH-812
>>
>> Best regards,
>> Alexandre
>>
>> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
>> > Hi,
>> >
>> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
>> > using similar commands. Can anyone help?
>> >
>> > Thanks,
>> > Yong
>> >
>> >
>> > 2013/12/6 Rohit Bhat <rohitbhat.nitk@gmail.com>
>> >
>> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
>> following 2
>> > > commands and got similar errors in both the cases:
>> > >
>> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
>> -DskipTests"
>> > >
>> > >
>> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>> > > package"
>> > >
>> > >
>> > > [INFO] Checking dependency versions
>> > >
>> > > [ERROR] Found a problem with the direct dependency
>> > > commons-httpclient:commons-httpclient of the current project
>> > >
>> > >   Expected version is 3.0.1
>> > >
>> > >   Resolved version is 3.0.1
>> > >
>> > >   Version 3.1 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
>> > >
>> > >   Resolved version is 2.4
>> > >
>> > >   Version 2.4 was expected by artifact:
>> > > commons-configuration:commons-configuration
>> > >
>> > >   Version 2.5 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency
>> > > commons-logging:commons-logging
>> > >
>> > >   Resolved version is 1.0.4
>> > >
>> > >   Version 1.1.1 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency io.netty:netty
>> > >
>> > >   Resolved version is 3.5.3.Final
>> > >
>> > >   Version 3.6.2.Final was expected by artifacts: org.apache.
>> > >
>> > > hadoop:hadoop-mapreduce-client-common, org.apache.
>> > > hadoop:hadoop-mapreduce-client-core
>> > >
>> > > [ERROR] Found a problem with the dependency
>> org.apache.zookeeper:zookeeper
>> > >
>> > >   Resolved version is 3.3.3
>> > >
>> > >   Version 3.4.5 was expected by artifacts:
>> > >   org.apache.hadoop:hadoop-common,
>> > >
>> > > org.apache.hadoop:hadoop-mapreduce-client-common
>> > >
>> > >
>> > > Am i missing something?
>> > >
>> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
>> <ameya.vilankar@gmail.com>wrote:
>> > >> Yes. Use this:
>> > >>
>> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>> > >>
>> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
>> <manuellagang@gmail.com>wrote:
>> > >>> I get the same error when I compile Giraph against the default
>> hadoop
>> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
>> > >>> recent hadoop version. Did you set the hadoop version via a maven
>> > >>> profile
>> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
>> 1.0)?
>> > >>>
>> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even
a
>> > >>> Yarn profile in the pom.xml. I personally haven't had success
>> building
>> > >>> it
>> > >>> though.
>> > >>>
>> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
>> <rohitbhat.nitk@gmail.com>wrote:
>> > >>>> Hi,
>> > >>>>
>> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and
have
>> > >>>> installed Giraph on top of it. I have been trying to
>> > >>>> execute SimpleShortestPathsComputation example but i am getting
the
>> > >>>> error:
>> > >>>>
>> > >>>> "java.lang.IllegalArgumentException:
>> checkLocalJobRunnerConfiguration:
>> > >>>> When using LocalJobRunner, you cannot run in split master /
worker
>> mode
>> > >>>> since there is only 1 task at a time!"
>> > >>>>
>> > >>>> I fixed this by adding an entry in mapred-site.xml for
>> > >>>> mapreduce.jobtracker.address but now i am getting a different
>> error:
>> > >>>>
>> > >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but
class
>> was
>> > >>>> expected*
>> > >>>>
>> > >>>>  at
>> > >>>>
>> > >>>>
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
>> > >>>> java:43) at
>> > >>>>
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
>> > >>>> 56)
>> > >>>>
>> > >>>>  at
>> > >>>>
>> > >>>>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
>> > >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>> > >>>>
>> > >>>>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>> > >>>>
>> > >>>> at java.security.AccessController.doPrivileged(Native Method)
>> > >>>>
>> > >>>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>> > >>>>
>> > >>>> at
>> > >>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> > >>>> on.java:1491)>>>>
>> > >>>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>> > >>>>
>> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> > >>>>
>> > >>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> > >>>>
>> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > >>>>
>> > >>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > >>>>
>> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> > >>>>
>> > >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > >>>>
>> > >>>> at
>> > >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> > >>>> va:57)>>>>
>> > >>>>  at
>> > >>>>
>> > >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
>> > >>>>
>> > >>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> > >>>>
>> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Rohit
>>
>
>

Mime
View raw message