giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Negro <alenegr...@yahoo.it>
Subject Re: Help with Giraph on Yarn
Date Thu, 20 Nov 2014 08:07:57 GMT
Hi,
this is mine:


ERROR:

yarn:/opt/yarn/hadoop-2.5.1/logs/userlogs/application_1416436887590_0001/container_1416436887590_0001_01_000001 />cat gam-stderr.log 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1416436887590_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-11-19 23:44:11.185 java[1082:1803] Unable to load realm info from SCDynamicStore
Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks.
	at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391)
	at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78)
	at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522)
	at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1416436887590_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
	at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153)
	at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
	at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387)
	... 6 more

OUTPUT:
yarn:/opt/yarn/hadoop-2.5.1/logs/userlogs/application_1416436887590_0001/container_1416436887590_0001_01_000001 />cat gam-stdout.log 
2014-11-19 23:44:10,651 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM 
2014-11-19 23:44:11,266 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-11-19 23:44:11,799 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId container_1416436887590_0001_01_000001 ApplicationAttemptId appattempt_1416436887590_0001_000001
2014-11-19 23:44:11,839 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at /0.0.0.0:8030
2014-11-19 23:44:11,866 INFO  [main] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool size is 500
2014-11-19 23:44:11,867 INFO  [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:<init>(78)) - yarn.client.max-nodemanagers-proxies : 500
2014-11-19 23:44:12,001 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-11-19 23:44:12,006 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-11-19 23:44:12,006 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
2014-11-19 23:44:14,042 INFO  [AMRM Heartbeater thread] impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - Received new token for : 192.168.0.13:49329
2014-11-19 23:44:14,044 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from RM for container ask, allocatedCnt=1
2014-11-19 23:44:14,044 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # of container so far : 1 allocated out of 2 required.
2014-11-19 23:44:14,044 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - Launching command on a new container., containerId=container_1416436887590_0001_01_000002, containerNode=192.168.0.13:49329, containerNodeURI=192.168.0.13:8042, containerResourceMemory=1024
2014-11-19 23:44:14,045 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up container launch container for containerid=container_1416436887590_0001_01_000002
2014-11-19 23:44:14,054 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} org.apache.giraph.yarn.GiraphYarnTask 1416436887590 1 2 1 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log 
2014-11-19 23:44:14,054 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting username in ContainerLaunchContext to: yarn
2014-11-19 23:44:14,347 INFO  [pool-4-thread-1] yarn.YarnUtils (YarnUtils.java:addFsResourcesToMap(72)) - Adding giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar to LocalResources for export.to hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1416436887590_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
2014-11-19 23:44:15,048 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from RM for container ask, allocatedCnt=1
2014-11-19 23:44:15,048 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # of container so far : 2 allocated out of 2 required.
2014-11-19 23:44:15,048 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - Launching command on a new container., containerId=container_1416436887590_0001_01_000003, containerNode=192.168.0.13:49329, containerNodeURI=192.168.0.13:8042, containerResourceMemory=1024
2014-11-19 23:44:15,049 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up container launch container for containerid=container_1416436887590_0001_01_000003
2014-11-19 23:44:15,049 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} org.apache.giraph.yarn.GiraphYarnTask 1416436887590 1 3 1 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log 
2014-11-19 23:44:15,049 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting username in ContainerLaunchContext to: yarn
2014-11-19 23:44:15,059 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event EventType: START_CONTAINER for Container container_1416436887590_0001_01_000003
2014-11-19 23:44:15,060 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy : 192.168.0.13:49329
2014-11-19 23:44:15,090 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event EventType: QUERY_CONTAINER for Container container_1416436887590_0001_01_000003
2014-11-19 23:44:16,053 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from RM for container ask, completedCnt=1
2014-11-19 23:44:16,054 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container status for containerID=container_1416436887590_0001_01_000003, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch: ExitCodeException exitCode=1: 
ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

2014-11-19 23:44:16,054 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(596)) - After completion of one conatiner. current status is: completedCount :1 containersToLaunch :2 successfulCount :0 failedCount :1
2014-11-19 23:54:47,775 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from RM for container ask, completedCnt=1
2014-11-19 23:54:47,775 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container status for containerID=container_1416436887590_0001_01_000002, state=COMPLETE, exitStatus=-100, diagnostics=Container expired since it was unused
2014-11-19 23:54:47,775 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(594)) - All container compeleted. done = true
2014-11-19 23:54:47,912 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(194)) - Done true
2014-11-19 23:54:47,912 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(198)) - Forcefully terminating executors with done =:true
2014-11-19 23:54:47,913 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(212)) - Application completed. Stopping running containers
2014-11-19 23:54:47,934 INFO  [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : 192.168.0.13:49329
2014-11-19 23:54:47,934 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(217)) - Application completed. Signalling finish to RM
2014-11-19 23:54:47,945 INFO  [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
2014-11-19 23:54:48,049 INFO  [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
2014-11-19 23:54:48,152 INFO  [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
2014-11-19 23:54:48,254 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(445)) - Giraph Application Master failed. exiting
2014-11-19 23:54:48,255 INFO  [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted while waiting for queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)



Il giorno 20/nov/2014, alle ore 08:15, Tripti Singh <tripti@yahoo-inc.com> ha scritto:

> Hi,
> For the infinite run time, could u share the container logs.
> I faced the similar issue and had to patch the GiraphApplicationMaster.
> If that is a common issue faced, I can submit my patch for review.
> 
> Thanks,
> Tripti.
> 
> From: Alessandro Negro <alenegro81@yahoo.it>
> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
> Date: Thursday, November 20, 2014 at 11:59 AM
> To: "user@giraph.apache.org" <user@giraph.apache.org>
> Subject: Re: Help with Giraph on Yarn
> 
> Hi,
> Now you should run it as yarn user. 
> 
> Try and let me know.
> 
> Alessandro
> 
> 
> 
> 
> Il giorno 20/nov/2014, alle ore 02:17, D Adams <dadamszx@gmail.com> ha scritto:
> 
>> Hi Allessandro,
>>      I made a copy of the giraph examples in my .../hadoop/share folder. This is more hopeful, but it is stuck in an infinite loop. I will try to do some searching regarding this ...
>> Running hduser script ...
>> bin/hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/17 12:49:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> 14/11/17 12:49:14 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
>> 14/11/17 12:49:14 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
>> 14/11/17 12:49:14 INFO yarn.GiraphYarnClient: Final output path is: hdfs://localhost:9000/user/hduser/output/shortestpaths
>> 14/11/17 12:49:14 INFO yarn.GiraphYarnClient: Running Client
>> 14/11/17 12:49:14 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8050
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=roosevelt:34304, nodeAddress roosevelt:8042, nodeRackName /default-rack, nodeNumContainers 0
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1416227900862_0012
>> 14/11/17 12:49:15 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: Set the environment for the application master
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>> 14/11/17 12:49:15 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>> 14/11/17 12:49:16 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416227900862_0012/giraph-conf.xml
>> 14/11/17 12:49:16 INFO yarn.GiraphYarnClient: LIB JARS :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/17 12:49:16 INFO yarn.YarnUtils: Class path name .
>> 14/11/17 12:49:16 INFO yarn.YarnUtils: base path checking .
>> 14/11/17 12:49:16 INFO yarn.GiraphYarnClient: Made local resource for :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar to hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416227900862_0012/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/17 12:49:20 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416227900862_0012/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/17 12:49:20 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
>> 14/11/17 12:49:20 INFO yarn.GiraphYarnClient: Submitting application to ASM
>> 14/11/17 12:49:20 INFO impl.YarnClientImpl: Submitted application application_1416227900862_0012
>> 14/11/17 12:49:20 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1416227900862_0012
>> 14/11/17 12:49:20 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
>> 14/11/17 12:49:21 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.85 secs
>> 14/11/17 12:49:21 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: ACCEPTED, Containers used: 1
>> 14/11/17 12:49:25 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.99 secs
>> 14/11/17 12:49:25 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: ACCEPTED, Containers used: 1
>> 14/11/17 12:49:29 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 9.01 secs
>> 14/11/17 12:49:29 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 1
>> 14/11/17 12:49:33 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 13.14 secs
>> 14/11/17 12:49:33 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:37 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 17.37 secs
>> 14/11/17 12:49:37 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:41 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 21.59 secs
>> 14/11/17 12:49:41 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:45 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 25.69 secs
>> 14/11/17 12:49:45 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:49 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 29.70 secs
>> 14/11/17 12:49:49 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:53 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 33.71 secs
>> 14/11/17 12:49:53 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:49:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 37.72 secs
>> 14/11/17 12:49:57 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:50:01 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 41.72 secs
>> 14/11/17 12:50:01 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:50:05 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 45.73 secs
>> 14/11/17 12:50:05 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:50:09 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 49.74 secs
>> 14/11/17 12:50:09 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0012_000001, State: RUNNING, Containers used: 3
>> 14/11/17 12:50:13 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 53.75 secs
>> ...
>> 
>> 
>> 
>> 
>> On Wed, Nov 19, 2014 at 5:04 PM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>> Could you try to copy this file: 
>>> 
>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar, giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>> 
>>> 
>>> into hadoop share library (in my case /opt/yarn/hadoop-2.5.1/share/myLib)
>>> 
>>> let me know
>>> 
>>> 
>>> Il giorno 19/nov/2014, alle ore 23:16, D Adams <dadamszx@gmail.com> ha scritto:
>>> 
>>>> Hi Alessandro,
>>>>      You were expecting Doug's response, but he is my output in the meanwhile:
>>>> 
>>>> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh 
>>>> Running hduser script ...
>>>> bin/hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>> 14/11/17 09:57:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>> 14/11/17 09:57:47 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
>>>> 14/11/17 09:57:47 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
>>>> 14/11/17 09:57:48 INFO yarn.GiraphYarnClient: Final output path is: hdfs://localhost:9000/user/hduser/output/shortestpaths
>>>> 14/11/17 09:57:48 INFO yarn.GiraphYarnClient: Running Client
>>>> 14/11/17 09:57:48 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8050
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=roosevelt:34304, nodeAddress roosevelt:8042, nodeRackName /default-rack, nodeNumContainers 0
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1416227900862_0011
>>>> 14/11/17 09:57:49 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: Set the environment for the application master
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>> 14/11/17 09:57:49 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>> 14/11/17 09:57:50 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416227900862_0011/giraph-conf.xml
>>>> 14/11/17 09:57:50 INFO yarn.GiraphYarnClient: LIB JARS :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>> 14/11/17 09:57:50 INFO yarn.YarnUtils: Class path name .
>>>> 14/11/17 09:57:50 INFO yarn.YarnUtils: base path checking .
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/etc/hadoop
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/etc/hadoop
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/common/lib/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/common/lib
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/common/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/common
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/hdfs
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/hdfs
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/hdfs/lib/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/hdfs/lib
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/hdfs/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/hdfs
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/yarn/lib/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/yarn/lib
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/yarn/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/yarn
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/mapreduce/lib/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/mapreduce/lib
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/share/hadoop/mapreduce/*
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/share/hadoop/mapreduce
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: Class path name /usr/local/hadoop/contrib/capacity-scheduler/*.jar
>>>> 14/11/17 09:57:51 INFO yarn.YarnUtils: base path checking /usr/local/hadoop/contrib/capacity-scheduler
>>>> 14/11/17 09:57:51 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
>>>> 14/11/17 09:57:51 INFO yarn.GiraphYarnClient: Submitting application to ASM
>>>> 14/11/17 09:57:51 INFO impl.YarnClientImpl: Submitted application application_1416227900862_0011
>>>> 14/11/17 09:57:51 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1416227900862_0011
>>>> 14/11/17 09:57:51 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
>>>> 14/11/17 09:57:52 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.92 secs
>>>> 14/11/17 09:57:52 INFO yarn.GiraphYarnClient: appattempt_1416227900862_0011_000001, State: ACCEPTED, Containers used: 1
>>>> 14/11/17 09:57:56 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1416227900862_0011 failed 2 times due to AM Container for appattempt_1416227900862_0011_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: 
>>>> ExitCodeException exitCode=1: 
>>>>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>     at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>     at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>>>>     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>     at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> 
>>>> Container exited with a non-zero exit code 1
>>>> .Failing this attempt.. Failing the application.
>>>> 14/11/17 09:57:56 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
>>>> 14/11/17 09:57:56 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 4 seconds.
>>>> Finished hduser script ...
>>>> 
>>>> And here is the log for the container:
>>>> Error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
>>>> 
>>>> 
>>>> On Wed, Nov 19, 2014 at 3:44 PM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>>>> Hi Doug,
>>>>> could you paste here the error from hadoop?
>>>>> 
>>>>> 
>>>>> Alessandro
>>>>> 
>>>>> Il giorno 19/nov/2014, alle ore 21:53, Doug Needham <doug.needham@ilwllc.com> ha scritto:
>>>>> 
>>>>>> I think I may be having a similar problem. Although it could be even more rudimentary than that.
>>>>>> I am running CDH5.2 cluster, RHEL 6.4
>>>>>> Giraph 1.1.0 (built form github today via: mvn -DskipTests -Dhadoop.version=2.3.0-cdh5.0.0 -Phadoop_yarn clean install)
>>>>>>  
>>>>>>  
>>>>>> Command I am running is:
>>>>>> hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.0-jar-with-dependencies.jar
>>>>>>  org.apache.giraph.GiraphRunner  org.apache.giraph.examples.SimplePageRankComputation   
>>>>>>  -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat       
>>>>>>  -vip /user/dln/ginput/tiny_graph.txt 
>>>>>>  -vof org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexOutputFormat 
>>>>>>  -op /user/dln/goutput2 -w 1
>>>>>>  
>>>>>> It runs for a few minutes then I get the following on stdout:
>>>>>>  
>>>>>> 14/11/19 15:48:16 INFO yarn.GiraphYarnClient: appattempt_1416424267079_0012_000002, State: ACCEPTED, Containers used: 0
>>>>>> 14/11/19 15:48:18 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimplePageRankComputation reports FAILED state, diagnostics show: Ap
>>>>>> plication application_1416424267079_0012 failed 2 times due to AM Container for appattempt_1416424267079_0012_000002 exited with  exitCode: 1 due to:
>>>>>> Exception from container-launch.
>>>>>> Container id: container_1416424267079_0012_02_000001
>>>>>> Exit code: 1
>>>>>> Stack trace: ExitCodeException exitCode=1:
>>>>>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>         at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>         at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>         at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
>>>>>>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>>>>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>         at java.lang.Thread.run(Thread.java:745)
>>>>>>  
>>>>>>  
>>>>>> Container exited with a non-zero exit code 1
>>>>>> .Failing this attempt.. Failing the application.
>>>>>> 14/11/19 15:48:18 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
>>>>>> 14/11/19 15:48:18 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimplePageRankComputation: FAILED, total running time: 0 mi
>>>>>> nutes, 14 seconds.
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>> Any suggestions?
>>>>>>  
>>>>>> Thank you,
>>>>>>  
>>>>>> Doug
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>> From: adamosdas@gmail.com [mailto:adamosdas@gmail.com] On Behalf Of D Adams
>>>>>> Sent: Wednesday, November 19, 2014 3:34 PM
>>>>>> To: user@giraph.apache.org
>>>>>> Subject: Re: Help with Giraph on Yarn
>>>>>>  
>>>>>> Alessandro,
>>>>>>      Sorry I didn't get back to you. I have 3 VMs right now and have gotten confused about which hadoop/giraph configuration I have on each. I will get back to this thread once I have regained awareness of which VM has that error. Thanks,
>>>>>> 
>>>>>> V/r
>>>>>> Das
>>>>>>  
>>>>>> On Wed, Nov 19, 2014 at 9:54 AM, Tripti Singh <tripti@yahoo-inc.com> wrote:
>>>>>>> Hi Alessandro,
>>>>>>>  
>>>>>>> I guess u r talking about the documentation on Giraph webpage which mentions addition of this new option which is more or less mandatory for yarn based profiles.
>>>>>>> When I first ran the Giraph on yarn without the proper –yj option, there was no way I could figure out from the Application/Container logs that this was the issue.
>>>>>>> I think, it’ll be a good idea to have this message in the logs for easy debugging.
>>>>>>>  
>>>>>>> Thanks,
>>>>>>> Tripti.
>>>>>>>  
>>>>>>>  
>>>>>>> From: Alessandro Negro <alenegro81@yahoo.it>
>>>>>>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>> Date: Tuesday, November 18, 2014 at 4:31 PM
>>>>>>> 
>>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>> Subject: Re: Help with Giraph on Yarn
>>>>>>>  
>>>>>>> Hi Eli,
>>>>>>> no I think that the help message is clear enough, generally when I read “jar” I always mean absolute path for jar, but this is a my personal misunderstanding.
>>>>>>>  
>>>>>>> Thanks,
>>>>>>> Alessandro
>>>>>>>  
>>>>>>>  
>>>>>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <apache.mailbox@gmail.com> ha scritto:
>>>>>>> 
>>>>>>> 
>>>>>>>> Sounds like you got it figured out since last time I checked this list, sorry it was a pain. Feel free to drop a JIRA ticket if we can make help message etc. for -yj option more clear, there's lots to improve there.
>>>>>>>>  
>>>>>>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>>>>>>>> Hi Tripti,
>>>>>>>>> finally I was able to run the test with success. It was an issue of permission since I was running as ale not as yarn.
>>>>>>>>> Let me say that now I’m able to run graph examples on Yarn 2.5.1. This is the final result:
>>>>>>>>>  
>>>>>>>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running time: 0 minutes, 21 seconds.
>>>>>>>>>  
>>>>>>>>>  
>>>>>>>>> Many thanks for your support,
>>>>>>>>> Alessandro
>>>>>>>>>  
>>>>>>>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> I don't know if u have access to this node. But if u do, u could check if the file is indeed there and u have access to it.
>>>>>>>>>> 
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>> 
>>>>>>>>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>>>>>>>> 
>>>>>>>>>>> You are right it works, but now I receive the following error:
>>>>>>>>>>>  
>>>>>>>>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>>>>>>>>> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>>>>>> SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>>>>>>>>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>>>>>>>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCDynamicStore
>>>>>>>>>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks.
>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391)
>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78)
>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522)
>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>>>>>>>>>>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>>>>>>>>>>> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153)
>>>>>>>>>>> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387)
>>>>>>>>>>> ... 6 more
>>>>>>>>>>>  
>>>>>>>>>>>  
>>>>>>>>>>> That justify the other error I receive in the task:
>>>>>>>>>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask
>>>>>>>>>>> 
>>>>>>>>>>>  
>>>>>>>>>>> Thanks,
>>>>>>>>>>>  
>>>>>>>>>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> Why r u adding two jars? Example jar ideally contains core library so everything should be available with just one example jar included
>>>>>>>>>>>> 
>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>> 
>>>>>>>>>>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> now it seems better, I need to add: 
>>>>>>>>>>>>>  
>>>>>>>>>>>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>>  
>>>>>>>>>>>>>  
>>>>>>>>>>>>> Now it seems that after a lot of cycle it fail with this error:
>>>>>>>>>>>>>  
>>>>>>>>>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask
>>>>>>>>>>>>>  
>>>>>>>>>>>>> But in this case the error appear in task-3-stderr.log  not in gam-stderr.log where there is the following error:
>>>>>>>>>>>>>  
>>>>>>>>>>>>> LF4J: Class path contains multiple SLF4J bindings.
>>>>>>>>>>>>> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>>>>>>>> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>>>>>>>> SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>>>>>>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>>>>>>>>>>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>>>>>>>>>>> 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDynamicStore
>>>>>>>>>>>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks.
>>>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391)
>>>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78)
>>>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522)
>>>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>>> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0006/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>>>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>>>>>>>>>>>>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>>>>>>>>>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>>>>>>>>>>>>> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153)
>>>>>>>>>>>>> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
>>>>>>>>>>>>> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387)
>>>>>>>>>>>>> ... 6 more
>>>>>>>>>>>>>  
>>>>>>>>>>>>>  
>>>>>>>>>>>>> Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> I remember seeing this thread, but since i was already deep into fixing this issue in production, I didn't really wait for the patch, not even the munge symbol as I was not even able to compile the yarn profile( sorry for being so ruthless to giraph :-( )
>>>>>>>>>>>>>> I did later cross check with the patch and it was quite similar to my change but sure it wouldn't work for all profiles(in fact some profiles may even break due to Hadoop class incompatibility ) I can share those changes with u once I get back to my computer but I don't think that could have caused this problem.
>>>>>>>>>>>>>> Just try the small changed I mentioned in my previous reply and let me know if that helped.
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Tripti 
>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Tripti,
>>>>>>>>>>>>>>> let me add one info, As you can see from this thread of email: 
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>> http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3CC3F75A98-BDE8-4205-971F-1F7241E6E83F@yahoo.it%3E
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>> I had some issue while compiling for Hadoop 2.5.1 could be that into the process I loose something?
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>> Thanks again,
>>>>>>>>>>>>>>> Alessandro
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>> Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegro81@yahoo.it> ha scritto:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Tripti,
>>>>>>>>>>>>>>>> yes I run in this way:
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> $HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1 -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> And this is the complete trace:
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from SCDynamicStore
>>>>>>>>>>>>>>>> 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=10.61.137.28:59677, nodeAddress 10.61.137.28:8042, nodeRackName /default-rack, nodeNumContainers 0
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415264041937_0005
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the application master
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name .
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking .
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>>>>>>>>>>> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs
>>>>>>>>>>>>>>>> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005_000001, State: ACCEPTED, Containers used: 1
>>>>>>>>>>>>>>>> 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415264041937_0005 failed 2 times due to AM Container for appattempt_1415264041937_0005_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>> ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> Container exited with a non-zero exit code 1
>>>>>>>>>>>>>>>> .Failing this attempt.. Failing the application.
>>>>>>>>>>>>>>>> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
>>>>>>>>>>>>>>>> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds.
>>>>>>>>>>>>>>>> ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_1415264041937_0005
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> As you can see there is this piece:
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> But I continue to receive the error:
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Alessandro
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>> Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> For GiraphApplicationMaster not found, the simple thing that worked for me was adding a -yj parameter to my command. 
>>>>>>>>>>>>>>>>> (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} {bunch of parameters} -yj {jar-file-name})
>>>>>>>>>>>>>>>>> If u look at job logs,
>>>>>>>>>>>>>>>>> Do u see a log statement defining "LIB JARS"?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>> yea my logs shows that error. I follow this example:
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C689.5060101@web.de%3E
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> without success.
>>>>>>>>>>>>>>>>>> Then I try to add all jars I can from giraph adding the following lines into the hadoop-env.sh:
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>>>>>>>>>>>>>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> when I run the application now I see this:
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> But again the same error. What I think is that there is no problem in the library but while is starting up.
>>>>>>>>>>>>>>>>>> I saw this piece in the logs:
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>>>>>>>>>>>>> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> That show that it is able to load GiraphApplicationMaster but fail during the startup.
>>>>>>>>>>>>>>>>>> Where can I find more logs?
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Alessandro
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>> Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Can u run the following command to find container logs and post it here:
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> yarn logs -applicationId application_1415203231170_0002
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> If u find this error:
>>>>>>>>>>>>>>>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>>>>>>>>>>>>>> It may just be due to non-availablity of the yarn jars
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Tripti
>>>>>>>>>>>>>>>>>>> Tech Yahoo
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> From: Jing Fan <fanjing09@gmail.com>
>>>>>>>>>>>>>>>>>>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>>>>>>>>>>>>>> Date: Thursday, November 6, 2014 at 1:56 AM
>>>>>>>>>>>>>>>>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>>>>>>>>>>>>>>>>> Subject: Re: Help with Giraph on Yarn
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> Then do you know to solve this error?
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch.
>>>>>>>>>>>>>>>>>>> Container id: container_1415203231170_0002_02_000001
>>>>>>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>>>>>>> Stack trace: ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>> On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>> in the mean time I found the reason why it hangs for long time here:
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>> base path checking .
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>> It scan all the directory tree starting from current directory, if you run it into your home (and you have a lot of files and directories) it takes ages to complete.
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>> I continue to got the same error.
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Alessandro
>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjing09@gmail.com> ha scritto:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I run with:
>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>> hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/jing/shortestpaths -w 1 -yj giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> This is my complete logs with the command I run:
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>> ale:~ />$HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1
>>>>>>>>>>>>>>>>>>>>>> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from SCDynamicStore
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=192.168.0.105:50658, nodeAddress192.168.0.105:8042, nodeRackName /default-rack, nodeNumContainers 0
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415189069846_0004
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the application master
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS :
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name .
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking .
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application application_1415189069846_0004
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415189069846_0004
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 secs
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415189069846_0004 failed 2 times due to AM Container for appattempt_1415189069846_0004_000002 exited with  exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>>>>>>>> ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>> Container exited with a non-zero exit code 1
>>>>>>>>>>>>>>>>>>>>>> .Failing this attempt.. Failing the application.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
>>>>>>>>>>>>>>>>>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds.
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <tripti@yahoo-inc.com> ha scritto:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I faced the similar problem where the workflow just hangs. 
>>>>>>>>>>>>>>>>>>>>>>> Can u see in ur logs if there is a task with id 0.
>>>>>>>>>>>>>>>>>>>>>>> I made a local fix and I could give u the patch if u see a similar problem and ur workflow is unable to find the zookeeper manager file and repeats the following message:
>>>>>>>>>>>>>>>>>>>>>>> For task xyz, got file 'null'
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Tripti
>>>>>>>>>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>> have you seen the hadoop logs?
>>>>>>>>>>>>>>>>>>>>>>>> Could you write here how are you running the command?
>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Alessandro
>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjing09@gmail.com> ha scritto:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Yeah, I add -yj option. The result is the same. Still error here.
>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <alenegro81@yahoo.it> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jing,
>>>>>>>>>>>>>>>>>>>>>>>>>> I’m facing the same problem. Do you check the hadoop logs? In my case I got: 
>>>>>>>>>>>>>>>>>>>>>>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>>>>>>>>>>>>>>>>>>>>> Moreover I don’t know why my application hang for a long time here:
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking .
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> And then continue failing with your error into the output and the other into the hadoop logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> Did you test adding this?
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Alessandro 
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjing09@gmail.com> ha scritto:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>> I install the Giraph with Yarn.
>>>>>>>>>>>>>>>>>>>>>>>>>>> But when I run the program with following command:
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>>  hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/j/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/j/shortestpaths -w 1
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>> The program exited with error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 secs
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 secs
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0
>>>>>>>>>>>>>>>>>>>>>>>>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Container id: container_1415203231170_0002_02_000001
>>>>>>>>>>>>>>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>>>>>>>>>>>>>>> Stack trace: ExitCodeException exitCode=1: 
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyone can help?
>>>>> 
>>>> 
>>> 
>> 


Mime
View raw message