giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Vesse <rve...@dotnetrdf.org>
Subject Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Date Mon, 18 Nov 2013 10:22:43 GMT
Hey all

Bumping this thread because I am still completely blocked on this and hoping
someone can help me.  As explained earlier in the thread I have Giraph built
for pure YARN on stable Hadoop 2.2.0 release like so:

mvn ­Phadoop_yarn ­Dhadoop.version=2.2.0 ­DskipTests=true clean install

But when I try and run the simple shortest paths demo like so:

$> bin/hadoop jar 
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
/user/rvesse/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/rvesse/shortestpaths -w 1 -yj
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar

The application fails and the YARN logs show the following error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.giraph.yarn.GiraphApplicationMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Any ideas why the GiraphApplicationMaster would not be found?  Have I just
got something basic wrong in my configuration or is this a bug in Giraph?

I found an earlier thread on the same topic
(http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D
1.40006@googlemail.com%3e) but it was never resolved hence why I have now
changed the topic in the hope it will jog someone's memory

Cheers,

Rob

From:  Rob Vesse <rvesse@dotnetrdf.org>
Reply-To:  <user@giraph.apache.org>
Date:  Friday, 15 November 2013 14:14
To:  <user@giraph.apache.org>
Subject:  Re: Giraph and Hadoop 2.2.0

> Adding the ­yj argument to specify the YARN JARs doesn't make any difference.
> 
> Here is the complete console output from my run:
> 
> bin/hadoop jar 
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example
> s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /user/rvesse/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/rvesse/shortestpaths/3 -w 1 -yj
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example
> s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
> SCDynamicStore
> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://localhost/user/rvesse/shortestpaths/3
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM for,
> nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
> nodeRackName/default-rack, nodeNumContainers0
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to
> 1024MB.
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1384523910015_0004
> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is deprecated.
> Instead, use mapreduce.job.id
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop
> /common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/shar
> e/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/sh
> are/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOM
> E/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources ::
> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0
> 004/giraph-conf.xml
> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exampl
> es-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> /contrib/capacity-scheduler/*.jar
> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> /contrib/capacity-scheduler
> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for
> GiraphApplicationMaster launch container is populated.
> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after submission
> :application_1384523910015_0004
> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82 secs
> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1384523910015_0004 failed 2
> times due to AM Container for appattempt_1384523910015_0004_000002 exited with
> exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchConta
> iner(DefaultContainerExecutor.java:195)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerL
> aunch.call(ContainerLaunch.java:283)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerL
> aunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java
> :895)
> at 
> 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918>
)
> at java.lang.Thread.run(Thread.java:695)
> 
> 
> .Failing this attempt.. Failing the application.
> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
> cache directory for Giraph job.
> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 3 seconds.
> 
> And I see the same error in the logs:
> 
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/giraph/yarn/GiraphApplicationMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.giraph.yarn.GiraphApplicationMaster
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> 
> Any more ideas?
> 
> Rob
> 
> From:  Mohammad Islam <mislam77@yahoo.com>
> Reply-To:  <user@giraph.apache.org>
> Date:  Thursday, 14 November 2013 20:36
> To:  "user@giraph.apache.org" <user@giraph.apache.org>
> Subject:  Re: Giraph and Hadoop 2.2.0
> 
>> I tried the following command few months back. I found you are not passing
>> some params.
>> 
>> bin/hadoop --config etc/hadoop jar
>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-
>> 1.?.0-for-hadoop-???-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip /user/me/input
>> -op /user/me/output_35w_20G_80m_1
>>  
>>  
>>  
>>  
>>  
>>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse <rvesse@dotnetrdf.org>
>> wrote:
>>   
>>   
>> It's the same command line I showed earlier, just trying to run the example
>> application:
>> 
>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exampl
>> es-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
>> /user/rvesse/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/rvesse/shortestpaths/2 -w 1
>> 
>> Rob
>> 
>> From:  Mohammad Islam <mislam77@yahoo.com>
>> Reply-To:  <user@giraph.apache.org>
>> Date:  Thursday, 14 November 2013 05:57
>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject:  Re: Giraph and Hadoop 2.2.0
>> 
>>> What are you running? PR?
>>> What is the command line?
>>> 
>>>  
>>>  
>>>  
>>>  
>>>  
>>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <rvesse@dotnetrdf.org>
>>> wrote:
>>>   
>>>   
>>> So I fixed the previous issue I was having which I reported and submitted a
>>> patch as GIRAPH-795
>>> 
>>> I then went ahead with installing and configuring my external Zookeeper
>>> appropriately and tried running again but now the YARN application fails and
>>> I find the following error in the logs:
>>> 
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>  org/apache/giraph/yarn/GiraphApplicationMaster
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> 
>>> Anybody have idea what the cause of this one might be or how to fix it?
>>> 
>>> Cheers,
>>> 
>>> Rob
>>> 
>>> From:  Rob Vesse <rvesse@dotnetrdf.org>
>>> Reply-To:  <user@giraph.apache.org>
>>> Date:  Wednesday, 13 November 2013 15:35
>>> To:  <user@giraph.apache.org>
>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>> 
>>>> Mohammad
>>>> 
>>>> This got me a build that at least attempts to start to run but not I see
>>>> the following:
>>>> 
>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exam
>>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/2 -w 1
>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>>> SCDynamicStore
>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
>>>> library for your platform... using builtin-java classes where applicable
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>> Exception in thread "main" java.lang.NullPointerException
>>>> at 
>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(Gir
>>>> aphYarnClient.java:460)
>>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
>>>> 9)
>>>> at 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>>>> l.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>> 
>>>> Which indirectly implies that I have some missing configuration to run
>>>> Giraph on YARN.  I don't think that should be a NPE and it looks to be a
>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>> 
>>>> The error message I should receive if the NPE wasn't happening is the
>>>> following:
>>>> 
>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances: use
a
>>>> standalone ZooKeeper.
>>>> 
>>>> What configuration options do I need to have set to point Giraph at an
>>>> external Zookeeper?
>>>> 
>>>> Cheers,
>>>> 
>>>> Rob
>>>> 
>>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>>> Reply-To:  <user@giraph.apache.org>
>>>> Date:  Wednesday, 13 November 2013 11:15
>>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>>> 
>>>>> Hi Rob,
>>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>>> clean package -DskipTests"?
>>>>> Regards,
>>>>> Mohammad
>>>>>  
>>>>>  
>>>>>  
>>>>>  
>>>>>  
>>>>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse
>>>>> <rvesse@dotnetrdf.org> wrote:
>>>>>   
>>>>>   
>>>>> Hi all
>>>>> 
>>>>> I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0
>>>>> installation.  I tried the hadoop_2.0.3 profile with no luck because
I get
>>>>> class cast errors when trying to run the examples.
>>>>> 
>>>>> Is there a specific profile I should be building to support Hadoop 2.2.0
>>>>> or is this not supported yet?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Rob
>>>>> 
>>>>> 
>>>>>   
>>>>>  
>>>>>   
>>>>>  
>>> 
>>> 
>>>   
>>>  
>>>   
>>>  
>> 
>> 
>>   
>>  
>>   
>>  



Mime
View raw message