giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Vesse <rve...@dotnetrdf.org>
Subject Re: Giraph and Hadoop 2.2.0
Date Fri, 15 Nov 2013 14:14:30 GMT
Adding the ­yj argument to specify the YARN JARs doesn't make any
difference.

Here is the complete console output from my run:

bin/hadoop jar 
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
/user/rvesse/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/rvesse/shortestpaths/3 -w 1 -yj
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
SCDynamicStore
13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
hdfs://localhost/user/rvesse/shortestpaths/3
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
nodeRackName/default-rack, nodeNumContainers0
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
to 1024MB.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application ID:
application_1384523910015_0004
13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
deprecated. Instead, use mapreduce.job.id
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the
application master
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
:{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hado
op/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/
share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_H
OME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MA
PRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduc
e/lib/*}
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015
_0004/giraph-conf.xml
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
:/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exam
ples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
/contrib/capacity-scheduler/*.jar
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
/contrib/capacity-scheduler
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
for GiraphApplicationMaster launch container is populated.
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to ASM
13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after submission
:application_1384523910015_0004
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
secs
13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1384523910015_0004 failed 2
times due to AM Container for appattempt_1384523910015_0004_000002 exited
with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
tainer(DefaultContainerExecutor.java:195)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
rLaunch.call(ContainerLaunch.java:283)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
rLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:895)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
18)
at java.lang.Thread.run(Thread.java:695)


.Failing this attempt.. Failing the application.
13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 3 seconds.

And I see the same error in the logs:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.giraph.yarn.GiraphApplicationMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Any more ideas?

Rob

From:  Mohammad Islam <mislam77@yahoo.com>
Reply-To:  <user@giraph.apache.org>
Date:  Thursday, 14 November 2013 20:36
To:  "user@giraph.apache.org" <user@giraph.apache.org>
Subject:  Re: Giraph and Hadoop 2.2.0

> I tried the following command few months back. I found you are not passing
> some params.
> 
> bin/hadoop --config etc/hadoop jar
> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1
> .?.0-for-hadoop-???-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimplePageRankVertexOpt -c
> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip /user/me/input
> -op /user/me/output_35w_20G_80m_1
>  
>  
>  
>  
>  
>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse <rvesse@dotnetrdf.org>
> wrote:
>   
>   
> It's the same command line I showed earlier, just trying to run the example
> application:
> 
> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example
> s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /user/rvesse/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/rvesse/shortestpaths/2 -w 1
> 
> Rob
> 
> From:  Mohammad Islam <mislam77@yahoo.com>
> Reply-To:  <user@giraph.apache.org>
> Date:  Thursday, 14 November 2013 05:57
> To:  "user@giraph.apache.org" <user@giraph.apache.org>
> Subject:  Re: Giraph and Hadoop 2.2.0
> 
>> What are you running? PR?
>> What is the command line?
>> 
>>  
>>  
>>  
>>  
>>  
>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <rvesse@dotnetrdf.org>
>> wrote:
>>   
>>   
>> So I fixed the previous issue I was having which I reported and submitted a
>> patch as GIRAPH-795
>> 
>> I then went ahead with installing and configuring my external Zookeeper
>> appropriately and tried running again but now the YARN application fails and
>> I find the following error in the logs:
>> 
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>  org/apache/giraph/yarn/GiraphApplicationMaster
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.giraph.yarn.GiraphApplicationMaster
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> 
>> Anybody have idea what the cause of this one might be or how to fix it?
>> 
>> Cheers,
>> 
>> Rob
>> 
>> From:  Rob Vesse <rvesse@dotnetrdf.org>
>> Reply-To:  <user@giraph.apache.org>
>> Date:  Wednesday, 13 November 2013 15:35
>> To:  <user@giraph.apache.org>
>> Subject:  Re: Giraph and Hadoop 2.2.0
>> 
>>> Mohammad
>>> 
>>> This got me a build that at least attempts to start to run but not I see the
>>> following:
>>> 
>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
>>> les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
>>> /user/rvesse/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/rvesse/shortestpaths/2 -w 1
>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>> SCDynamicStore
>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
>>> library for your platform... using builtin-java classes where applicable
>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>> specified. Ensure your OutputFormat does not require one.
>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>> Exception in thread "main" java.lang.NullPointerException
>>> at 
>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(Gira
>>> phYarnClient.java:460)
>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at 
>>> 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39>>>
)
>>> at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>> .java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>> 
>>> Which indirectly implies that I have some missing configuration to run
>>> Giraph on YARN.  I don't think that should be a NPE and it looks to be a
>>> trivial fix so I'll try and fix and send a pull request for that.
>>> 
>>> The error message I should receive if the NPE wasn't happening is the
>>> following:
>>> 
>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances: use a
>>> standalone ZooKeeper.
>>> 
>>> What configuration options do I need to have set to point Giraph at an
>>> external Zookeeper?
>>> 
>>> Cheers,
>>> 
>>> Rob
>>> 
>>> From:  Mohammad Islam <mislam77@yahoo.com>
>>> Reply-To:  <user@giraph.apache.org>
>>> Date:  Wednesday, 13 November 2013 11:15
>>> To:  "user@giraph.apache.org" <user@giraph.apache.org>
>>> Subject:  Re: Giraph and Hadoop 2.2.0
>>> 
>>>> Hi Rob,
>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>> clean package -DskipTests"?
>>>> Regards,
>>>> Mohammad
>>>>  
>>>>  
>>>>  
>>>>  
>>>>  
>>>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <rvesse@dotnetrdf.org>
>>>> wrote:
>>>>   
>>>>   
>>>> Hi all
>>>> 
>>>> I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0
>>>> installation.  I tried the hadoop_2.0.3 profile with no luck because I get
>>>> class cast errors when trying to run the examples.
>>>> 
>>>> Is there a specific profile I should be building to support Hadoop 2.2.0
or
>>>> is this not supported yet?
>>>> 
>>>> Thanks,
>>>> 
>>>> Rob
>>>> 
>>>> 
>>>>   
>>>>  
>>>>   
>>>>  
>> 
>> 
>>   
>>  
>>   
>>  
> 
> 
>   
>  
>   
>  



Mime
View raw message