giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Mishra <mishravis...@gmail.com>
Subject Re: Issue while executing SimpleShortestPathsComputation
Date Mon, 09 Dec 2013 20:45:17 GMT
Tried the steps (one variation was 'clean package' instead of 'compile'
since a dependency check was failing) provided by Alexandre and hitting the
same error as Rohit.
Note that I'm attempting on a cluster.

Cmdline: hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/rrdata/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/rrdata/output -w 1 -yj
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
-ca giraph.SplitMasterWorker=false -ca giraph.zkList=exotica:2181




On Mon, Dec 9, 2013 at 10:34 AM, Rohit Bhat <rohitbhat.nitk@gmail.com>wrote:

> Hi Alexandre,
>
> I looked into the logs and found this:
>
> 2013-12-09 12:29:08,925 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(426)) - Starting GitaphAM
> 2013-12-09 12:29:09,727 WARN  [main] util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
> for your$
> 2013-12-09 12:29:10,206 ERROR [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught a
> t$
> java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/
> *TextOutputFormat*
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:788)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> In gam-stderr.log i found this:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/scratch/rohit/hadoop-2.2.0/tmp/nm-local-dir/usercache/rohit/appcache/application_1386613677359_0001/filecach$
> SLF4J: Found binding in
> [jar:file:/scratch/rohit/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinde$
>
> I tried the fix that you mentioned of linking and adding -yj for hadoop-
> mapreduce-client-core-2.2.0.jar but that did not help. HADOOP_MAPRED_HOME
> is pointing to HADOOP_HOME on my setup.
>
>
> On Mon, Dec 9, 2013 at 10:11 AM, Alexandre Fonseca <
> alexandrejorgefonseca@gmail.com> wrote:
>
>> Hi Rohit,
>>
>> In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
>> SimpleShortestPathsComputation:
>>
>> - Error: Could not find or load main class
>> org.apache.giraph.yarn.GiraphApplicationMaster
>>
>>   This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar
>>
>> - java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
>>
>>    This was solved by making sure HADOOP_MAPRED_HOME was
>> pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and
>> restarting
>> both the nodemanager and resourcemanager. Another thing that also worked
>> was
>> linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
>> core-2.2.0.jar file to the current directory and adding it to the -yj
>> parameter:
>>         -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
>> dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
>>
>> I found the first error on:
>>
>> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_
>> 1386443083254_0001_01_000001/gam-stderr.log
>>
>> And the second error on:
>>
>> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_
>> 1386443083254_0001_01_000002/gam-stdout.log
>>
>> If you didn't change the default yarn property yarn.nodemanager.log-dirs,
>> you
>> should be able to see them on the respective directories on your side. If
>> not,
>> check on whatever you set that property to.
>>
>> Hope this helps!
>>
>> Alex
>>
>> On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
>> > Hi Eli/Alexandre,
>> >
>> > I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using
>> the
>> > steps mentioned by Alexandre, but again i am getting the same error
>> while
>> > running the example:
>> >
>> > Command:
>> >
>> > $HADOOP_HOME/bin/hadoop jar
>> >
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> > org.apache.giraph.GiraphRunner
>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > -vip /giraphInput/tiny_graph.txt -vof
>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
>> > giraph.zkList=localhost:2181 -yj
>> >
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> >
>> > ERROR yarn.GiraphYarnClient: Giraph:
>> > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>> > state, diagnostics show: Application application_1386601013698_0005
>> failed
>> > 2 times due to AM Container for appattempt_1386601013698_0005_000002
>> exited
>> > with  exitCode: 1 due to: Exception from container-launch:
>> > org.apache.hadoop.util.Shell$ExitCodeException:
>> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
>> > at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>> > at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
>> > tainer(DefaultContainerExecutor.java:195) at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
>> > rLaunch.call(ContainerLaunch.java:283) at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
>> > rLaunch.call(ContainerLaunch.java:79) at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
>> > java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
>> > 45) at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
>> > 15) at java.lang.Thread.run(Thread.java:724)
>> >
>> >
>> > .Failing this attempt.. Failing the application.
>> >
>> > I checked the Hadoop logs as well but couldn't find anything meaningful
>> > there too.
>> >
>> > Thanks,
>> > Rohit
>> >
>> > On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
>> <apache.mailbox@gmail.com>wrote:
>> > > Hey all,
>> > >
>> > > So. I think from more recent emails Rohit got this figured out, but, a
>> > > couple points just so they out there again:
>> > >
>> > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
>> > > 2.0.03-alpha or (with a patch thats still floating around on
>> GIRAPH-687
>> > > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
>> > >
>> > > Much better way to go is to build against Hadoop 2.2.x line using
>> Giraph
>> > > 1.1.x's YARN profile. This features a stable API, some basic support
>> for
>> > > security, and a concurrency fix that could bite you when launching
>> many
>> > > workers.
>> > >
>> > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need
>> to
>> > > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
>> > > profile.
>> > > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x
>> do,
>> > > but riding on Mapper task instead of running the workers in YARN
>> > > containers.
>> > >
>> > >
>> > >
>> > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
>> > >
>> > > alexandrejorgefonseca@gmail.com> wrote:
>> > >> Hello Yong,
>> > >>
>> > >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
>> > >> hadoop-2.2.0.
>> > >>
>> > >> Here's what I did:
>> > >>         git clone https://git-wip-us.apache.org/repos/asf/giraph.git
>> > >>         cd giraph
>> > >>         git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
>> > >>         mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>> > >>
>> > >> With that I got valid jars which allowed me to run the
>> > >> SimpleShortestPaths
>> > >>
>> > >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>> > >>         mkdir test_example
>> > >>         ln -s
>> > >>         $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
>> > >>
>> > >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> > >>
>> > >>         $HADOOP_PREFIX/bin/hadoop jar
>> giraph-examples-1.1.0-SNAPSHOT-for-
>> > >>
>> > >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
>> > >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > >>
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > >> -vip
>> > >> giraph/input/tiny_graph.txt -vof
>> > >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false
>> -ca
>> > >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> > >> hadoop-2.2.0-jar-with-dependencies.jar
>> > >>
>> > >> ###########
>> > >> Notes:
>> > >> * Wasn't able to do mvn package as it complained about not having the
>> > >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is
>> just a
>> > >> matter
>> > >> of changing the main pom.xml to include those modules with the hadoop
>> > >> 2.2.0
>> > >> target or remove those dependencies from the giraph-dist if they no
>> > >> longer
>> > >> apply.
>> > >> * You may get some memory limit exceeded problems for the
>> > >> ApplicationMaster.
>> > >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio
>> parameter in
>> > >> yarn-
>> > >> site.xml or apply my patch to prevent allocation of heap size equal
>> to
>> > >> total
>> > >> memory size, described at:
>> > >> https://issues.apache.org/jira/browse/GIRAPH-812
>> > >> * In my case, the application didn't exit even after outputting the
>> > >> result and
>> > >> having recognized that all nodes voted to terminate. I've opened an
>> issue
>> > >> in JIRA regarding this so if you suffer from the same, you can apply
>> that
>> > >> patch
>> > >> too: https://issues.apache.org/jira/browse/GIRAPH-811
>> > >>
>> > >> ###########
>> > >> Relevant JIRA issues:
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-798
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-804
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-811
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-812
>> > >>
>> > >> Best regards,
>> > >> Alexandre
>> > >>
>> > >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
>> > >> > Hi,
>> > >> >
>> > >> > I also got this error when tried to build Giraph on top of
>> hadoop-2.2.0
>> > >> > using similar commands. Can anyone help?
>> > >> >
>> > >> > Thanks,
>> > >> > Yong
>> > >> >
>> > >> >
>> > >> > 2013/12/6 Rohit Bhat <rohitbhat.nitk@gmail.com>
>> > >> >
>> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried
the
>> > >>
>> > >> following 2
>> > >>
>> > >> > > commands and got similar errors in both the cases:
>> > >> > >
>> > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
>> > >>
>> > >> -DskipTests"
>> > >>
>> > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean
>> install
>> > >> > > package"
>> > >> > >
>> > >> > >
>> > >> > > [INFO] Checking dependency versions
>> > >> > >
>> > >> > > [ERROR] Found a problem with the direct dependency
>> > >> > > commons-httpclient:commons-httpclient of the current project
>> > >> > >
>> > >> > >   Expected version is 3.0.1
>> > >> > >
>> > >> > >   Resolved version is 3.0.1
>> > >>
>> > >> > >   Version 3.1 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency
>> commons-lang:commons-lang
>> > >> > >
>> > >> > >   Resolved version is 2.4
>> > >> > >
>> > >> > >   Version 2.4 was expected by artifact:
>> > >> > > commons-configuration:commons-configuration
>> > >> > >
>> > >> > >   Version 2.5 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency
>> > >> > > commons-logging:commons-logging
>> > >> > >
>> > >> > >   Resolved version is 1.0.4
>> > >>
>> > >> > >   Version 1.1.1 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency io.netty:netty
>> > >> > >
>> > >> > >   Resolved version is 3.5.3.Final
>> > >> > >
>> > >> > >   Version 3.6.2.Final was expected by artifacts: org.apache.
>> > >> > >
>> > >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
>> > >> > > hadoop:hadoop-mapreduce-client-core
>> > >> > >
>> > >> > > [ERROR] Found a problem with the dependency
>> > >>
>> > >> org.apache.zookeeper:zookeeper
>> > >>
>> > >> > >   Resolved version is 3.3.3
>> > >> > >
>> > >> > >   Version 3.4.5 was expected by artifacts:
>> > >> > >   org.apache.hadoop:hadoop-common,
>> > >> > >
>> > >> > > org.apache.hadoop:hadoop-mapreduce-client-common
>> > >> > >
>> > >> > >
>> > >> > > Am i missing something?
>> > >> > >
>> > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
>> > >>
>> > >> <ameya.vilankar@gmail.com>wrote:
>> > >> > >> Yes. Use this:
>> > >> > >>
>> > >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true
>> package
>> > >> > >>
>> > >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
>> > >>
>> > >> <manuellagang@gmail.com>wrote:
>> > >> > >>> I get the same error when I compile Giraph against
the default
>> > >>
>> > >> hadoop
>> > >>
>> > >> > >>> version (0.20.203.0), but my project that uses Giraph
uses a
>> more
>> > >> > >>> recent hadoop version. Did you set the hadoop version
via a
>> maven
>> > >> > >>> profile
>> > >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile
for hadoop
>> > >>
>> > >> 1.0)?
>> > >>
>> > >> > >>> Presumably, there's a way to compile Giraph for Yarn.
There's
>> even
>> > >> > >>> a
>> > >> > >>> Yarn profile in the pom.xml. I personally haven't
had success
>> > >>
>> > >> building
>> > >>
>> > >> > >>> it
>> > >> > >>> though.
>> > >> > >>>
>> > >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
>> > >>
>> > >> <rohitbhat.nitk@gmail.com>wrote:
>> > >> > >>>> Hi,
>> > >> > >>>>
>> > >> > >>>> I am new to Giraph. I have installed Yarn (hadoop
2.2.0) and
>> have
>> > >> > >>>> installed Giraph on top of it. I have been trying
to
>> > >> > >>>> execute SimpleShortestPathsComputation example
but i am
>> getting
>> > >> > >>>> the
>> > >> > >>>> error:
>> > >>
>> > >> > >>>> "java.lang.IllegalArgumentException:
>> > >> checkLocalJobRunnerConfiguration:
>> > >> > >>>> When using LocalJobRunner, you cannot run in
split master /
>> worker
>> > >>
>> > >> mode
>> > >>
>> > >> > >>>> since there is only 1 task at a time!"
>> > >> > >>>>
>> > >> > >>>> I fixed this by adding an entry in mapred-site.xml
for
>> > >> > >>>> mapreduce.jobtracker.address but now i am getting
a different
>> > >>
>> > >> error:
>> > >> > >>>> *Exception in thread "main"
>> > >> > >>>> java.lang.IncompatibleClassChangeError:
>> > >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext,
but
>> class
>> > >>
>> > >> was
>> > >>
>> > >> > >>>> expected*
>> > >> > >>>>
>> > >> > >>>>  at
>> > >>
>> > >>
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
>> > >>
>> > >> > >>>> java:43) at
>> > >>
>> > >>
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
>> > >>
>> > >> > >>>> 56)
>> > >> > >>>>
>> > >> > >>>>  at
>> > >>
>> > >>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
>> > >>
>> > >> > >>>> .java:342) at
>> > >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>> > >> > >>>>
>> > >> > >>>>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>> > >> > >>>>
>> > >> > >>>> at java.security.AccessController.doPrivileged(Native
Method)
>> > >> > >>>>
>> > >> > >>>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> > >>
>> > >> > >>>> on.java:1491)>>>>
>> > >> > >>>>
>> > >> > >>>>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>> > >> > >>>>
>> > >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> > >> > >>>>
>> > >> > >>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > >> > >>>>
>> > >> > >>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > >> > >>>>
>> > >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> > >> > >>>>
>> > >> > >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> > >>
>> > >> > >>>> va:57)>>>>
>> > >> > >>>>
>> > >> > >>>>  at
>> > >>
>> > >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> > >>
>> > >> > >>>> rImpl.java:43) at
>> java.lang.reflect.Method.invoke(Method.java:606)
>> > >> > >>>>
>> > >> > >>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> > >> > >>>>
>> > >> > >>>> *Is Giraph compatible with yarn? If yes, am i
missing
>> anything?*
>> > >> > >>>>
>> > >> > >>>> Thanks,
>> > >> > >>>> Rohit
>>
>
>

Mime
View raw message