giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Vesse <rve...@dotnetrdf.org>
Subject Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Date Mon, 09 Dec 2013 11:42:03 GMT
Yes it actually is in the JAR file:

mbp-rvesse:target rvesse$ jar -tf
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep
org.apache.giraph.yarn.GiraphApplicationMaster
org/apache/giraph/yarn/GiraphApplicationMaster$1.class
org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster.class

Rob

From:  Rafal Wojdyla <ravwojdyla@gmail.com>
Reply-To:  <user@giraph.apache.org>
Date:  Monday, 9 December 2013 10:24
To:  <user@giraph.apache.org>
Subject:  Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
2.2.0)

> Hi all!
> 
> I had the same problems as you guys:
> 
> @Rob:
> Did you check if org.apache.giraph.yarn.GiraphApplicationMaster is actually in
> the jar file - it wasn't it my case - I'm using custom profile in maven.
> Do: jar -tf 
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example
> s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep
> org.apache.giraph.yarn.GiraphApplicationMaster
> 
> @Vishal:
> I got this error few times when cluster was highly utilized, problem
> disappeared when I used queue with free resources - this may be a problem but
> I haven't investigated it.
> 
> Unfortunately I wasn't able to successfully run Giraph application anyway, due
> to: https://issues.apache.org/jira/browse/GIRAPH-806
> 
> Cheers - Rafal
> 
> 
> On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <apache.mailbox@gmail.com> wrote:
>> Hi all,
>> 
>> It looks to me like you want to run the giraph-core jar at the beginning of
>> the "hadoop jar" call and supply the giraph-examples jar with -yj argument.
>> 
>> Also, you can place any -D type configuration settings you're sick of
>> supplying at the command line into giraph-site.xml just as you would with
>> Hadoop settings in one of its own site.xml files, and place these where your
>> local Hadoop client configs are and they should be picked up in your
>> application.
>> 
>> Hope that helps,
>> 
>> Eli
>> 
>> 
>> 
>> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mishravishal@gmail.com>
>> wrote:
>>> I had a mistake in my command (moronic copy/paste zookeeper settings error).
>>> Updated and now hitting the following:
>>> 
>>> ....
>>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>>> master/192.168.1.100:8045 <http://192.168.1.100:8045>
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
>>> nodeId=kreator:46477, nodeAddresskreator:8042, nodeRackName/default-rack,
>>> nodeNumContainers7
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
>>> nodeId=exotica:46645, nodeAddressexotica:8042, nodeRackName/default-rack,
>>> nodeNumContainers8
>>> Exception in thread "main" java.lang.RuntimeException: Giraph job requires 2
>>> containers to run; cluster only hosts 15
>>> at 
>>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(Girap
>>> hYarnClient.java:230)
>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>>> ....
>>> 
>>> It seems like some config is missing from our hadoop setup.
>>> Also, note that I'm working from the trunk branch.
>>> 
>>> Thanx,
>>> Vishal
>>> 
>>> 
>>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mishravishal@gmail.com>
>>> wrote:
>>>> Hey Guys,
>>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
>>>> same problem as outlined in this thread. Was there a resolution? If yes,
>>>> can you please share?
>>>> Completely new to giraph and hadoop so please excuse any omissions. Our
>>>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>> 
>>>> Command:
>>>> hadoop jar 
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado
>>>> op-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -D
>>>> giraph.zkList="zkNode.net:2081"
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/hduser/rrdata/output -w 1 -yj
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado
>>>> op-2.2.0-jar-with-dependencies.jar
>>>> 
>>>> Trimmed Console Output:
>>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>>> with  exitCode: 1 due to: Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at 
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at 
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCo
>>>> ntainer(DefaultContainerExecutor.java:195)
>>>> at 
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain
>>>> erLaunch.call(ContainerLaunch.java:283)
>>>> at 
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain
>>>> erLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
>>>> 145)
>>>> at 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
>>>> 615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 
>>>> 
>>>> Thanx,
>>>> Vishal
>>>> 
>>>> 
>>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mislam77@yahoo.com>
wrote:
>>>>> Rob,
>>>>> I understand your frustration.
>>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>>> If no one replied by then, I will recreate the situation here and try
to
>>>>> give you more concrete help.
>>>>> 
>>>>> Regards,
>>>>> Mohammad
>>>>>  
>>>>>  
>>>>>  
>>>>>  
>>>>>  
>>>>>   On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <rvesse@dotnetrdf.org>
>>>>> wrote:
>>>>>   
>>>>>   
>>>>> Mohammad
>>>>> 
>>>>> Comments inline:
>>>>> 
>>>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>>>> Reply-To:  <user@giraph.apache.org>
>>>>> Date:  Monday, 18 November 2013 22:24
>>>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>>>> Subject:  Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>>>>> 2.2.0)
>>>>> 
>>>>>> I used the following shell script to run PR where my current working
>>>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>>>> . You can try something similar.
>>>>> 
>>>>> I already am.
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>>>> hadoop fs -rmr out
>>>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>>>> #export 
>>>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-had
>>>>>> oop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> #export 
>>>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>>>> SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> 
>>>>>> 
>>>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>> 
>>>>> What is the point of putting the conf directory on the CLASSPATH?  I
>>>>> assume this is only to pick up the giraph configuration but at the same
>>>>> time there is nothing to stop me putting my Giraph specific config in
one
>>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>> 
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>>> 
>>>>>> export CLASSPATH=$CLASSPATH
>>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>>> 
>>>>>> echo $HADOOP_CLASSPATH
>>>>> 
>>>>> I assume the point of the above is to ensure that the Giraph config file
>>>>> is on the class path Hadoop actually uses but adding this makes no
>>>>> difference to me, I get exactly the same failure as before.  Modifying
the
>>>>> logic to add the examples JAR onto the class path does not help either.
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependenc
>>>>>> ies.jar  org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependenc
>>>>>> ies.jar -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMaste
>>>>>> rCompute -vip input -op out
>>>>> 
>>>>> This is not the same command that I am running, also DoubleSumCombiner
>>>>> gives a ClassNotFoundException, you actually want
>>>>> DoubleSumMessageCombiner.  If I adapt my script to run your command I
>>>>> experience the exact same error as I do trying to run the command I showed
>>>>> so I am still nowhere.
>>>>> 
>>>>> I'm getting a little frustrated that no-one has any idea what the problem
>>>>> is here, either this is a simple misconfiguration on my part or a bug
in
>>>>> Giraph for Hadoop 2.2.0 ?  Given the thread so far and the fact that
I can
>>>>> run MR based Giraph on Hadoop 1.2.1 this is looking increasingly unlikely
>>>>> I'm starting to lean heavily towards the latter option.
>>>>> 
>>>>> Rob
>>>>> 
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>>   On Monday, November 18, 2013 2:24 AM, Rob Vesse <rvesse@dotnetrdf.org>
>>>>>> wrote:
>>>>>>   
>>>>>>   
>>>>>> Hey all
>>>>>> 
>>>>>> Bumping this thread because I am still completely blocked on this
and
>>>>>> hoping someone can help me.  As explained earlier in the thread I
have
>>>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>>> 
>>>>>> mvn ­Phadoop_yarn ­Dhadoop.version=2.2.0 ­DskipTests=true clean
install
>>>>>> 
>>>>>> But when I try and run the simple shortest paths demo like so:
>>>>>> 
>>>>>> $> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-ex
>>>>>> amples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-ex
>>>>>> amples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> 
>>>>>> The application fails and the YARN logs show the following error:
>>>>>> 
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>> 
>>>>>> Any ideas why the GiraphApplicationMaster would not be found?  Have
I
>>>>>> just got something basic wrong in my configuration or is this a bug
in
>>>>>> Giraph?
>>>>>> 
>>>>>> I found an earlier thread on the same topic
>>>>>> (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5
>>>>>> A4D1.40006@googlemail.com%3e) but it was never resolved hence why
I have
>>>>>> now changed the topic in the hope it will jog someone's memory
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Rob
>>>>>> 
>>>>>> From:  Rob Vesse <rvesse@dotnetrdf.org>
>>>>>> Reply-To:  <user@giraph.apache.org>
>>>>>> Date:  Friday, 15 November 2013 14:14
>>>>>> To:  <user@giraph.apache.org>
>>>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>>>> 
>>>>>>> Adding the ­yj argument to specify the YARN JARs doesn't make
any
>>>>>>> difference.
>>>>>>> 
>>>>>>> Here is the complete console output from my run:
>>>>>>> 
>>>>>>> bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm
info from
>>>>>>> SCDynamicStore
>>>>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java
classes
>>>>>>> where applicable
>>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input
format
>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output
format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path
is:
>>>>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager
at
>>>>>>> /0.0.0.0:8032 <http://0.0.0.0:8032>
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report
from ASM
>>>>>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>>>>> nodeRackName/default-rack, nodeNumContainers0
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task
heap
>>>>>>> size to 1024MB.
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>>>>>> ID: application_1384523910015_0004
>>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id
>>>>>>> <http://mapred.job.id>  is deprecated. Instead, use mapreduce.job.id
>>>>>>> <http://mapreduce.job.id>
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment
for
>>>>>>> the application master
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for
AM
>>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/
>>>>>>> hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HD
>>>>>>> FS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$H
>>>>>>> ADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/
>>>>>>> lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/s
>>>>>>> hare/hadoop/mapreduce/lib/*}
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap
....
>>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir
is
>>>>>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources
>>>>>>> :: 
>>>>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_138452391
>>>>>>> 0015_0004/giraph-conf.xml
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
>>>>>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /contrib/capacity-scheduler/*.jar
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /contrib/capacity-scheduler
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch
>>>>>>> container is populated.
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application
to
>>>>>>> ASM
>>>>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>>>>> <http://0.0.0.0:8032>
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>>>>> submission :application_1384523910015_0004
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
0.82
>>>>>>> secs
>>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers
used:
>>>>>>> 1
>>>>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports
FAILED
>>>>>>> state, diagnostics show: Application application_1384523910015_0004
>>>>>>> failed 2 times due to AM Container for
>>>>>>> appattempt_1384523910015_0004_000002 exited with  exitCode: 1
due to:
>>>>>>> Exception from container-launch:
>>>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>>>> at 
>>>>>>> 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589>>>>>>>
)
>>>>>>> at 
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc
>>>>>>> hContainer(DefaultContainerExecutor.java:195)
>>>>>>> at 
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
>>>>>>> ainerLaunch.call(ContainerLaunch.java:283)
>>>>>>> at 
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
>>>>>>> ainerLaunch.call(ContainerLaunch.java:79)
>>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>> at 
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
>>>>>>> r.java:895)
>>>>>>> at 
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>>>>> va:918)
>>>>>>> at java.lang.Thread.run(Thread.java:695)
>>>>>>> 
>>>>>>> 
>>>>>>> .Failing this attempt.. Failing the application.
>>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>>>> distributed cache directory for Giraph job.
>>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED,
total
>>>>>>> running time: 0 minutes, 3 seconds.
>>>>>>> 
>>>>>>> And I see the same error in the logs:
>>>>>>> 
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>> 
>>>>>>> Any more ideas?
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>>>>>> Reply-To:  <user@giraph.apache.org>
>>>>>>> Date:  Thursday, 14 November 2013 20:36
>>>>>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>>>>> 
>>>>>>> I tried the following command few months back. I found you are
not
>>>>>>> passing some params.
>>>>>>> 
>>>>>>> bin/hadoop --config etc/hadoop jar
>>>>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-exam
>>>>>>> ples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480
-yj
>>>>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar
-vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-vip
>>>>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse
>>>>>>> <rvesse@dotnetrdf.org> wrote:
>>>>>>>   
>>>>>>>   
>>>>>>> It's the same command line I showed earlier, just trying to run
the
>>>>>>> example application:
>>>>>>> 
>>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>>>>>> Reply-To:  <user@giraph.apache.org>
>>>>>>> Date:  Thursday, 14 November 2013 05:57
>>>>>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>>>>> 
>>>>>>> What are you running? PR?
>>>>>>> What is the command line?
>>>>>>> 
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse
>>>>>>> <rvesse@dotnetrdf.org> wrote:
>>>>>>>   
>>>>>>>   
>>>>>>> So I fixed the previous issue I was having which I reported and
>>>>>>> submitted a patch as GIRAPH-795
>>>>>>> 
>>>>>>> I then went ahead with installing and configuring my external
Zookeeper
>>>>>>> appropriately and tried running again but now the YARN application
fails
>>>>>>> and I find the following error in the logs:
>>>>>>> 
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>  org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>> 
>>>>>>> Anybody have idea what the cause of this one might be or how
to fix it?
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> From:  Rob Vesse <rvesse@dotnetrdf.org>
>>>>>>> Reply-To:  <user@giraph.apache.org>
>>>>>>> Date:  Wednesday, 13 November 2013 15:35
>>>>>>> To:  <user@giraph.apache.org>
>>>>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>>>>> 
>>>>>>> Mohammad
>>>>>>> 
>>>>>>> This got me a build that at least attempts to start to run but
not I see
>>>>>>> the following:
>>>>>>> 
>>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm
info from
>>>>>>> SCDynamicStore
>>>>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java
classes
>>>>>>> where applicable
>>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input
format
>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output
format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path
is:
>>>>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>>> at 
>>>>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(
>>>>>>> GiraphYarnClient.java:460)
>>>>>>> at 
>>>>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at 
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>>>>>>> a:39)
>>>>>>> at 
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>>>>> Impl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>> 
>>>>>>> Which indirectly implies that I have some missing configuration
to run
>>>>>>> Giraph on YARN.  I don't think that should be a NPE and it looks
to be a
>>>>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>>>>> 
>>>>>>> The error message I should receive if the NPE wasn't happening
is the
>>>>>>> following:
>>>>>>> 
>>>>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>>>>> use a standalone ZooKeeper.
>>>>>>> 
>>>>>>> What configuration options do I need to have set to point Giraph
at an
>>>>>>> external Zookeeper?
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>>>>>> Reply-To:  <user@giraph.apache.org>
>>>>>>> Date:  Wednesday, 13 November 2013 11:15
>>>>>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>>>>> 
>>>>>>> Hi Rob,
>>>>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>>>>> clean package -DskipTests"?
>>>>>>> Regards,
>>>>>>> Mohammad
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>  
>>>>>>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse
>>>>>>> <rvesse@dotnetrdf.org> wrote:
>>>>>>>   
>>>>>>>   
>>>>>>> Hi all
>>>>>>> 
>>>>>>> I'm having trouble getting Giraph to build and run against a
Hadoop
>>>>>>> 2.2.0 installation.  I tried the hadoop_2.0.3 profile with no
luck
>>>>>>> because I get class cast errors when trying to run the examples.
>>>>>>> 
>>>>>>> Is there a specific profile I should be building to support Hadoop
2.2.0
>>>>>>> or is this not supported yet?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> 
>>>>>>>   
>>>>>>>  
>>>>>>>   
>>>>>>>  
>>>>>>> 
>>>>>>> 
>>>>>>>   
>>>>>>>  
>>>>>>>   
>>>>>>>  
>>>>>>> 
>>>>>>> 
>>>>>>>   
>>>>>>>  
>>>>>>>   
>>>>>>>  
>>>>>> 
>>>>>> 
>>>>>>   
>>>>>>  
>>>>>>   
>>>>>>  
>>>>> 
>>>>> 
>>>>>   
>>>>>  
>>>>>   
>>>>>  
>>>> 
>>> 
>> 
> 



Mime
View raw message