giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafal Wojdyla <ravwojd...@gmail.com>
Subject Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Date Tue, 17 Dec 2013 10:45:44 GMT
@Vishal - please try: https://issues.apache.org/jira/browse/GIRAPH-819


On Mon, Dec 9, 2013 at 9:56 PM, Vishal Mishra <mishravishal@gmail.com>wrote:

> Rafal, thanx for pointing that out earlier and it was the problem since we
> were running a fairly intensive job but re-trying got me back to the
> previous error.
>
> Eli,
> I tried your suggestion but it leads to java.lang.ClassNotFoundException
> for SimpleShortestPathsComputation as it's not defined there. Maybe I
> didn't understand your suggestion?
> Thanx for the pointer to add the external zookeeper setting to
> giraph-site.xml esp. since following Alexandre Fonseca's suggestion leads
> to different command line options.
>
> Alex, I've replied to the other thread where I'm hitting the same error as
> Rohit.
>
>
>
> On Mon, Dec 9, 2013 at 3:42 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
>
>> Yes it actually is in the JAR file:
>>
>> mbp-rvesse:target rvesse$ jar -tf
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> | grep org.apache.giraph.yarn.GiraphApplicationMaster
>> org/apache/giraph/yarn/GiraphApplicationMaster$1.class
>>
>> org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
>> org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
>> org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
>> org/apache/giraph/yarn/GiraphApplicationMaster.class
>>
>> Rob
>>
>> From: Rafal Wojdyla <ravwojdyla@gmail.com>
>> Reply-To: <user@giraph.apache.org>
>> Date: Monday, 9 December 2013 10:24
>> To: <user@giraph.apache.org>
>>
>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>> 2.2.0)
>>
>> Hi all!
>>
>> I had the same problems as you guys:
>>
>> @Rob:
>> Did you check if *org.apache.giraph.yarn.GiraphApplicationMaster* is actually
>> in the jar file - it wasn't it my case - I'm using custom profile in maven.
>> Do: *jar -tf
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> | grep org.apache.giraph.yarn.GiraphApplicationMaster*
>>
>> @Vishal:
>> I got this error few times when cluster was highly utilized, problem
>> disappeared when I used queue with free resources - this may be a problem
>> but I haven't investigated it.
>>
>> Unfortunately I wasn't able to successfully run Giraph application
>> anyway, due to: https://issues.apache.org/jira/browse/GIRAPH-806
>>
>> Cheers - Rafal
>>
>>
>> On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> It looks to me like you want to run the giraph-core jar at the beginning
>>> of the "hadoop jar" call and supply the giraph-examples jar with -yj
>>> argument.
>>>
>>> Also, you can place any -D type configuration settings you're sick of
>>> supplying at the command line into giraph-site.xml just as you would with
>>> Hadoop settings in one of its own site.xml files, and place these where
>>> your local Hadoop client configs are and they should be picked up in your
>>> application.
>>>
>>> Hope that helps,
>>>
>>> Eli
>>>
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mishravishal@gmail.com>wrote:
>>>
>>>> I had a mistake in my command (moronic copy/paste zookeeper settings
>>>> error).
>>>> Updated and now hitting the following:
>>>>
>>>> ....
>>>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>>>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>>>> master/192.168.1.100:8045
>>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=kreator:46477, nodeAddresskreator:8042,
>>>> nodeRackName/default-rack, nodeNumContainers7
>>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=exotica:46645, nodeAddressexotica:8042,
>>>> nodeRackName/default-rack, nodeNumContainers8
>>>> Exception in thread "main" java.lang.RuntimeException: Giraph job
>>>> requires 2 containers to run; cluster only hosts 15
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>>>>  at
>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>>>> ....
>>>>
>>>> It seems like some config is missing from our hadoop setup.
>>>> Also, note that I'm working from the trunk branch.
>>>>
>>>> Thanx,
>>>> Vishal
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mishravishal@gmail.com
>>>> > wrote:
>>>>
>>>>> Hey Guys,
>>>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting
>>>>> the same problem as outlined in this thread. Was there a resolution?
If
>>>>> yes, can you please share?
>>>>> Completely new to giraph and hadoop so please excuse any omissions.
>>>>> Our cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>>>
>>>>> Command:
>>>>> hadoop jar
>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/hduser/rrdata/output -w 1 -yj
>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>
>>>>> Trimmed Console Output:
>>>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used:
1
>>>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002
exited
>>>>> with  exitCode: 1 due to: Exception from container-launch:
>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>>  at
>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>  at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>>>  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>>
>>>>> Thanx,
>>>>> Vishal
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mislam77@yahoo.com>wrote:
>>>>>
>>>>>> Rob,
>>>>>> I understand your frustration.
>>>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>>>> If no one replied by then, I will recreate the situation here and
try
>>>>>> to give you more concrete help.
>>>>>>
>>>>>> Regards,
>>>>>> Mohammad
>>>>>>
>>>>>>
>>>>>>   On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>>   Mohammad
>>>>>>
>>>>>> Comments inline:
>>>>>>
>>>>>> From: Mohammad Islam <mislam77@yahoo.com>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Monday, 18 November 2013 22:24
>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and
>>>>>> Hadoop 2.2.0)
>>>>>>
>>>>>> I used the following shell script to run PR where my current working
>>>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>>>> . You can try something similar.
>>>>>>
>>>>>>
>>>>>> I already am.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>>>> hadoop fs -rmr out
>>>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>>>> #export
>>>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> #export
>>>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>>
>>>>>>
>>>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>>>
>>>>>>
>>>>>> What is the point of putting the conf directory on the CLASSPATH?
 I
>>>>>> assume this is only to pick up the giraph configuration but at the
same
>>>>>> time there is nothing to stop me putting my Giraph specific config
in one
>>>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>>>
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>>>
>>>>>> export CLASSPATH=$CLASSPATH
>>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>>>
>>>>>> echo $HADOOP_CLASSPATH
>>>>>>
>>>>>>
>>>>>> I assume the point of the above is to ensure that the Giraph config
>>>>>> file is on the class path Hadoop actually uses but adding this makes
no
>>>>>> difference to me, I get exactly the same failure as before.  Modifying
the
>>>>>> logic to add the examples JAR onto the class path does not help either.
>>>>>>
>>>>>>
>>>>>>
>>>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>>  org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>>>>> -vip input -op out
>>>>>>
>>>>>>
>>>>>> This is not the same command that I am running, also
>>>>>> DoubleSumCombiner gives a ClassNotFoundException, you actually want
>>>>>> DoubleSumMessageCombiner.  If I adapt my script to run your command
I
>>>>>> experience the exact same error as I do trying to run the command
I showed
>>>>>> so I am still nowhere.
>>>>>>
>>>>>> I'm getting a little frustrated that no-one has any idea what the
>>>>>> problem is here, either this is a simple misconfiguration on my part
or a
>>>>>> bug in Giraph for Hadoop 2.2.0 ?  Given the thread so far and the
fact that
>>>>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>>>>> unlikely I'm starting to lean heavily towards the latter option.
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>>
>>>>>>   On Monday, November 18, 2013 2:24 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>>  Hey all
>>>>>>
>>>>>> Bumping this thread because I am still completely blocked on this
and
>>>>>> hoping someone can help me.  As explained earlier in the thread I
have
>>>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>>>
>>>>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean
>>>>>> install
>>>>>>
>>>>>> But when I try and run the simple shortest paths demo like so:
>>>>>>
>>>>>> $> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>
>>>>>> The application fails and the YARN logs show the following error:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Any ideas why the GiraphApplicationMaster would not be found?  Have
I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>>>>
>>>>>>
>>>>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e)
but it was never resolved hence why I have now changed the topic in the hope it will jog someone's
memory
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Rob Vesse <rvesse@dotnetrdf.org>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Friday, 15 November 2013 14:14
>>>>>> To: <user@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>>>>> difference.
>>>>>>
>>>>>> Here is the complete console output from my run:
>>>>>>
>>>>>> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info
>>>>>> from SCDynamicStore
>>>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes
where
>>>>>> applicable
>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output
>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager
>>>>>> at /0.0.0.0:8032
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from
>>>>>> ASM for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>>>> nodeRackName/default-rack, nodeNumContainers0
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task
>>>>>> heap size to 1024MB.
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new
>>>>>> Application ID: application_1384523910015_0004
>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>>>>> deprecated. Instead, use mapreduce.job.id
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment
for
>>>>>> the application master
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap
>>>>>> ....
>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir
>>>>>> is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
>>>>>> LocalResources ::
>>>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /contrib/capacity-scheduler/*.jar
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /contrib/capacity-scheduler
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>>>>> is populated.
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application
>>>>>> to ASM
>>>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>>>> submission :application_1384523910015_0004
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
0.82
>>>>>> secs
>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers
used: 1
>>>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports
FAILED
>>>>>> state, diagnostics show: Application application_1384523910015_0004
failed
>>>>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002
exited
>>>>>> with  exitCode: 1 due to: Exception from container-launch:
>>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>>>  at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>>> at
>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>>>>  at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>>>>  at
>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>  at java.lang.Thread.run(Thread.java:695)
>>>>>>
>>>>>>
>>>>>> .Failing this attempt.. Failing the application.
>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>>> distributed cache directory for Giraph job.
>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED,
total
>>>>>> running time: 0 minutes, 3 seconds.
>>>>>>
>>>>>> And I see the same error in the logs:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Any more ideas?
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Mohammad Islam <mislam77@yahoo.com>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Thursday, 14 November 2013 20:36
>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> I tried the following command few months back. I found you are not
>>>>>> passing some params.
>>>>>>
>>>>>> bin/hadoop --config etc/hadoop jar
>>>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-of
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>>>
>>>>>>
>>>>>>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>>  It's the same command line I showed earlier, just trying to run
the
>>>>>> example application:
>>>>>>
>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> From: Mohammad Islam <mislam77@yahoo.com>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Thursday, 14 November 2013 05:57
>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> What are you running? PR?
>>>>>> What is the command line?
>>>>>>
>>>>>>
>>>>>>
>>>>>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>>  So I fixed the previous issue I was having which I reported and
>>>>>> submitted a patch as GIRAPH-795
>>>>>>
>>>>>> I then went ahead with installing and configuring my external
>>>>>> Zookeeper appropriately and tried running again but now the YARN
>>>>>> application fails and I find the following error in the logs:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>  org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>>>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Anybody have idea what the cause of this one might be or how to fix
it?
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Rob Vesse <rvesse@dotnetrdf.org>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Wednesday, 13 November 2013 15:35
>>>>>> To: <user@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Mohammad
>>>>>>
>>>>>> This got me a build that at least attempts to start to run but not
I
>>>>>> see the following:
>>>>>>
>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info
>>>>>> from SCDynamicStore
>>>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes
where
>>>>>> applicable
>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output
>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>>  at
>>>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>>>>> at
>>>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>  at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>
>>>>>> Which indirectly implies that I have some missing configuration to
>>>>>> run Giraph on YARN.  I don't think that should be a NPE and it looks
to be
>>>>>> a trivial fix so I'll try and fix and send a pull request for that.
>>>>>>
>>>>>> The error message I should receive if the NPE wasn't happening is
the
>>>>>> following:
>>>>>>
>>>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>>>> use a standalone ZooKeeper.
>>>>>>
>>>>>> What configuration options do I need to have set to point Giraph
at
>>>>>> an external Zookeeper?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> From: Mohammad Islam <mislam77@yahoo.com>
>>>>>> Reply-To: <user@giraph.apache.org>
>>>>>> Date: Wednesday, 13 November 2013 11:15
>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Hi Rob,
>>>>>> Did you try with this command "mvn -Phadoop_yarn
>>>>>> -Dhadoop.version=2.2.0 clean package -DskipTests"?
>>>>>> Regards,
>>>>>> Mohammad
>>>>>>
>>>>>>
>>>>>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>>  Hi all
>>>>>>
>>>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>>>> 2.2.0 installation.  I tried the hadoop_2.0.3 profile with no luck
because
>>>>>> I get class cast errors when trying to run the examples.
>>>>>>
>>>>>> Is there a specific profile I should be building to support Hadoop
>>>>>> 2.2.0 or is this not supported yet?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message