giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From D Adams <dadam...@gmail.com>
Subject Re: Help with Giraph on Yarn
Date Sun, 23 Nov 2014 14:28:19 GMT
Well, maybe my giraph build did not build correctly because when I run

2014-11-19 02:04:21,764 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
2014-11-19 02:04:23,137 WARN  [main] util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
2014-11-19 02:04:25,548 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId
container_1416359131664_0002_01_000001 ApplicationAttemptId
appattempt_1416359131664_0002_000001
2014-11-19 02:04:25,890 INFO  [main] client.RMProxy
(RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
localhost/127.0.0.1:8030
2014-11-19 02:04:25,924 INFO  [main] impl.NMClientAsyncImpl
(NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread
pool size is 500
2014-11-19 02:04:25,925 INFO  [main]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:<init>(78)) -
yarn.client.max-nodemanagers-proxies : 500
2014-11-19 02:04:26,309 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:3000, vCores:0>]Priority[10]
2014-11-19 02:04:26,577 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:3000, vCores:0>]Priority[10]
2014-11-19 02:04:26,577 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(185)) - Wait to finish ..
2014-11-19 02:04:28,612 INFO  [AMRM Heartbeater thread]
impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) -
Received new token for : roosevelt:58195
2014-11-19 02:04:28,614 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
response from RM for container ask, allocatedCnt=1
2014-11-19 02:04:28,614 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
allocated # of container so far : 1 allocated out of 2 required.
2014-11-19 02:04:28,614 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1416359131664_0002_01_000002,
containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
containerResourceMemory=3072
2014-11-19 02:04:28,615 INFO  [pool-4-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
Setting up container launch container for
containerid=container_1416359131664_0002_01_000002
2014-11-19 02:04:28,629 INFO  [pool-4-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
Conatain launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1416359131664 2 2 1
1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
2014-11-19 02:04:28,630 INFO  [pool-4-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
Setting username in ContainerLaunchContext to: hduser
2014-11-19 02:04:29,149 INFO  [pool-4-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(72)) - Adding
giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar to
LocalResources for export.to
hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
2014-11-19 02:04:29,188 INFO  [pool-4-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar*2014-11-19
02:04:29,188 WARN  [pool-4-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
didn't include giraph-core.*
2014-11-19 02:04:29,189 INFO  [pool-4-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-conf.xml
2014-11-19 02:04:29,201 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: START_CONTAINER for Container
container_1416359131664_0002_01_000002
2014-11-19 02:04:29,202 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy
: roosevelt:58195
2014-11-19 02:04:29,257 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: QUERY_CONTAINER for Container
container_1416359131664_0002_01_000002
2014-11-19 02:04:29,618 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
response from RM for container ask, allocatedCnt=1
2014-11-19 02:04:29,618 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
allocated # of container so far : 2 allocated out of 2 required.
2014-11-19 02:04:29,618 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1416359131664_0002_01_000003,
containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
containerResourceMemory=3072
2014-11-19 02:04:29,619 INFO  [pool-4-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
Setting up container launch container for
containerid=container_1416359131664_0002_01_000003
2014-11-19 02:04:29,619 INFO  [pool-4-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
Conatain launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1416359131664 2 3 1
1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
2014-11-19 02:04:29,619 INFO  [pool-4-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
Setting username in ContainerLaunchContext to: hduser
2014-11-19 02:04:29,620 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: START_CONTAINER for Container
container_1416359131664_0002_01_000003
2014-11-19 02:04:29,623 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: QUERY_CONTAINER for Container
container_1416359131664_0002_01_000003
2014-11-19 02:04:30,620 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(571)) - Got
response from RM for container ask, completedCnt=2
2014-11-19 02:04:30,620 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got
container status for
containerID=container_1416359131664_0002_01_000002, state=COMPLETE,
exitStatus=1, diagnostics=Exception from container-launch:
ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2014-11-19 02:04:30,620 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got
container status for
containerID=container_1416359131664_0002_01_000003, state=COMPLETE,
exitStatus=1, diagnostics=Exception from container-launch:
ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2014-11-19 02:04:30,620 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(594)) - All
container compeleted. done = true
2014-11-19 02:04:30,791 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(194)) - Done true
2014-11-19 02:04:30,791 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(198)) - Forcefully terminating
executors with done =:true
2014-11-19 02:04:30,791 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(212)) - Application completed.
Stopping running containers
2014-11-19 02:04:30,803 INFO  [main]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing
proxy : roosevelt:58195
2014-11-19 02:04:30,803 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(217)) - Application completed.
Signalling finish to RM
2014-11-19 02:04:30,815 INFO  [main] impl.AMRMClientImpl
(AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for
application to be successfully unregistered.
2014-11-19 02:04:30,923 INFO  [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) -
Interrupted while waiting for queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
2014-11-19 02:04:30,944 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(445)) - Giraph Application Master
failed. exiting

In a moment, I will paste all the logs I can find.

On Sun, Nov 23, 2014 at 1:11 AM, Tripti Singh <tripti@yahoo-inc.com> wrote:

>   Sorry, the e-mail was sent without me being able to complete it.
>  The issue I was facing was due to a Giraph’s hard coded dependency on a
> task with id 0. This task is the designated Master responsible for creating
> a zookeeperServerList.
>  These task ids are assigned using ContainerId as input and with the
> assumption that we will be assigned containers starting with id of 2 (1 for
> Application Master)
>  Following is the comment available in Giraph Yarn class
>
>  /**
>
>    * Utility to create a TaskAttemptId we can feed to our fake
> Mapper#Context.
>
>    *
>
>    * NOTE: ContainerId will serve as MR TaskID for Giraph tasks.
>
>    * YARN container 1 is always AppMaster, so the least container id we
> will
>
>    * ever get from YARN for a Giraph task is container id 2. Giraph on
> MapReduce
>
>    * tasks must start at index 0. So we SUBTRACT TWO from each container
> id.
>
>    *
>
>    * @param args the command line args, fed to us by
> GiraphApplicationMaster.
>
>    * @return the TaskAttemptId object, populated with YARN job data.
>
>    */
>
>
>  Now, my problem was that in our cluster, this guarantee was not met and
> at times, I was not assigned the Container with id of 2 (And container ids
> are also not assigned in increasing order).
>
> Due to this, the Zookeeper was not starting in many workflow runs.
>
> The fix I did, was to get rid of this issue.
>
>
>  Thanks,
>  Tripti
>    From: Tripti Singh <tripti@yahoo-inc.com>
> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
> Date: Sunday, November 23, 2014 at 12:23 PM
>
> To: "user@giraph.apache.org" <user@giraph.apache.org>
> Subject: Re: Help with Giraph on Yarn
>
>   Hi Das,
> Could u try what Alessandro is suggesting. Giraph-example jar has all the
> classes required..
> Also, you have shared only AM logs. Can u also add the worker logs?
> From a quick look at the AM, I’m not sure if this is the same problem that
> I faced.
>
>  Thanks,
>  Tripti
>    From: Alessandro Negro <alenegro81@yahoo.it>
> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
> Date: Sunday, November 23, 2014 at 5:10 AM
> To: "user@giraph.apache.org" <user@giraph.apache.org>
> Subject: Re: Help with Giraph on Yarn
>
>   HI Das,
> what about the hadoop logs? Have a look there and let me know.
> In the meantime you can remove redundant jars from yarn jars (exampe
> contains all jars needed):
>
>   yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>
>
>  Should be enough
>
>  Alessandro.
>
>   Il giorno 23/nov/2014, alle ore 00:16, D Adams <dadamszx@gmail.com> ha
> scritto:
>
>    Hi Alessandro,
>       Do you run tyarn that way because your hadoop or giraph file
> permissions are chowned by a user named yarn?
>
>  I have made changes as suggested on
> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C689.5060101@web.de%3E
>
>  And although things do look more hopeful, I still get an infinite loop. I
> run with as script which includes:
>
> user_dir=/user/hduser
> jar=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> runner=org.apache.giraph.GiraphRunner
> computation=org.apache.giraph.examples.SimpleShortestPathsComputation
>
> informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>
> yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar
>
> bin/hdfs dfs -rm -r $user_dir/output/shortestpaths
> bin/yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000
> $computation -vif $informat -vip $user_dir/input/tiny_graph.txt -vof
> $outformat -op $user_dir/output/shortestpaths -w 1 -yj $yarnjars
>
>  And the resulting stdout:
> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh
> Running hduser script ...
> 14/11/18 19:05:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: `/user/hduser/output/shortestpaths': No such file or directory
> 14/11/18 19:06:13 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://localhost:9000/user/hduser/output/shortestpaths
> 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Running Client
> 14/11/18 19:06:16 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8050
> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Got node report from ASM
> for, nodeId=roosevelt:58195, nodeAddress roosevelt:8042, nodeRackName
> /default-rack, nodeNumContainers 0
> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1416359131664_0001
> 14/11/18 19:06:17 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 14/11/18 19:06:19 INFO yarn.YarnUtils: Registered file in LocalResources
> ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-conf.xml
> 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: LIB JARS
> :giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar
> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name .
> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking .
> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name /usr/local/hadoop
> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking /usr/local/hadoop
> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name null
> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking null
> 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: Made local resource for
> :/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> to
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/18 19:06:23 INFO yarn.YarnUtils: Registered file in LocalResources
> ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
> for GiraphApplicationMaster launch container is populated.
> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/11/18 19:06:23 INFO impl.YarnClientImpl: Submitted application
> application_1416359131664_0001
> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Got new appId after
> submission :application_1416359131664_0001
> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 14/11/18 19:06:24 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.90
> secs
> 14/11/18 19:06:24 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 0
> 14/11/18 19:06:28 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.05
> secs
> 14/11/18 19:06:28 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 19:06:33 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 9.89
> secs
> 14/11/18 19:06:33 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 19:06:37 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.37
> secs
> 14/11/18 19:06:37 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 19:06:42 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.81
> secs
> 14/11/18 19:06:42 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 19:06:46 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 22.82
> secs
> 14/11/18 19:06:46 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:06:50 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 26.83
> secs
> 14/11/18 19:06:50 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:06:54 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 30.83
> secs
> 14/11/18 19:06:54 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:06:58 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 34.85
> secs
> 14/11/18 19:06:58 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:07:02 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 38.85
> secs
> 14/11/18 19:07:02 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:07:06 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 42.86
> secs
> 14/11/18 19:07:06 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:07:10 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 46.88
> secs
> 14/11/18 19:07:10 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:07:14 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 50.89
> secs
> 14/11/18 19:07:14 INFO yarn.GiraphYarnClient:
> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
> 14/11/18 19:07:18 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 54.90
> secs
>
>  Once yarn hits the RUNNING state, it enters an endless loop. It seems it
> is caused by the following:
>
>
>  2014-11-18 19:06:37,382 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
> 2014-11-18 19:06:39,214 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2014-11-18 19:06:41,438 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId container_1416359131664_0001_01_000001 ApplicationAttemptId appattempt_1416359131664_0001_000001
> 2014-11-18 19:06:41,497 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at localhost/127.0.0.1:8030
> 2014-11-18 19:06:41,581 INFO  [main] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool size is 500
> 2014-11-18 19:06:41,583 INFO  [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:<init>(78)) - yarn.client.max-nodemanagers-proxies : 500
> 2014-11-18 19:06:42,355 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested container ask: Capability[<memory:3000, vCores:0>]Priority[10]
> 2014-11-18 19:06:42,370 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested container ask: Capability[<memory:3000, vCores:0>]Priority[10]
> 2014-11-18 19:06:42,618 INFO  [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
> 2014-11-18 19:06:44,673 INFO  [AMRM Heartbeater thread] impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - Received new token for : roosevelt:58195
> 2014-11-18 19:06:44,677 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from RM for container ask, allocatedCnt=1
> 2014-11-18 19:06:44,677 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # of container so far : 1 allocated out of 2 required.
> 2014-11-18 19:06:44,678 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - Launching command on a new container., containerId=container_1416359131664_0001_01_000002, containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042, containerResourceMemory=3072
> 2014-11-18 19:06:44,679 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up container launch container for containerid=container_1416359131664_0001_01_000002
> 2014-11-18 19:06:44,694 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH} org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 2 1 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
> 2014-11-18 19:06:44,694 INFO  [pool-4-thread-1] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting username in ContainerLaunchContext to: hduser
> 2014-11-18 19:06:45,191 INFO  [pool-4-thread-1] yarn.YarnUtils (YarnUtils.java:addFsResourcesToMap(72)) - Adding giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar to LocalResources for export.to hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> 2014-11-18 19:06:45,228 INFO  [pool-4-thread-1] yarn.YarnUtils (YarnUtils.java:addFileToResourceMap(160)) - Registered file in LocalResources :: hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> 2014-11-18 19:06:45,228 INFO  [pool-4-thread-1] yarn.YarnUtils (YarnUtils.java:addFsResourcesToMap(72)) - Adding giraph-core-1.1.0-SNAPSHOT.jar to LocalResources for export.to hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-core-1.1.0-SNAPSHOT.jar
> 2014-11-18 19:06:45,683 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from RM for container ask, allocatedCnt=1
> 2014-11-18 19:06:45,683 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # of container so far : 2 allocated out of 2 required.
> 2014-11-18 19:06:45,683 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - Launching command on a new container., containerId=container_1416359131664_0001_01_000003, containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042, containerResourceMemory=3072
> 2014-11-18 19:06:45,684 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up container launch container for containerid=container_1416359131664_0001_01_000003
> 2014-11-18 19:06:45,684 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH} org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 3 1 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
> 2014-11-18 19:06:45,684 INFO  [pool-4-thread-3] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting username in ContainerLaunchContext to: hduser
> 2014-11-18 19:06:45,697 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event EventType: START_CONTAINER for Container container_1416359131664_0001_01_000003
> 2014-11-18 19:06:45,699 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy : roosevelt:58195
> 2014-11-18 19:06:45,761 INFO  [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event EventType: QUERY_CONTAINER for Container container_1416359131664_0001_01_000003
> 2014-11-18 19:06:46,687 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from RM for container ask, completedCnt=1
> 2014-11-18 19:06:46,688 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container status for containerID=container_1416359131664_0001_01_000003, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch: ExitCodeException exitCode=1:
> ExitCodeException exitCode=1:
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:455)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> 	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> 2014-11-18 19:06:46,688 INFO  [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(596)) - After completion of one conatiner. current status is: completedCount :1 containersToLaunch :2 successfulCount :0 failedCount :1
>
>
> On Sat, Nov 22, 2014 at 11:42 AM, D Adams <dadamszx@gmail.com> wrote:
>
>>  I'm sorry, please excuse my ignorance, but, I have no idea where to
>> find my configuration. That is, I have no idea where these lines came from
>> (or what file):
>>
>> yarn             1029   0,1  1,2  3499572 209560 s001  S    10:29am
>> 0:06.04 /Library/Java/
>> JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>> -Dproc_resourcemanager -Xmx500m
>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>> ...
>>  As far as configuration goes, I'm only really aware of the
>> etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above
>> part of the command you use to run giraph jobs?
>>  I will make changes to hadoop-env.sh file as you suggest and let you
>> know how it works out.
>>
>> On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro <alenegro81@yahoo.it>
>> wrote:
>>
>>> Hi,
>>> I forgot to say that I added also this line:
>>>
>>>  export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>>>
>>>
>>>  Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro <
>>> alenegro81@yahoo.it> ha scritto:
>>>
>>>  Hi Das,
>>> this is my configuration of hadoop:
>>>
>>>  yarn             1029   0,1  1,2  3499572 209560 s001  S    10:29am
>>> 0:06.04
>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>>> -Dproc_resourcemanager -Xmx500m
>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
>>> -Dyarn.root.logger=INFO,RFA
>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
>>> -Dyarn.policy.file=hadoop-policy.xml
>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
>>> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
>>> -Dyarn.root.logger=INFO,RFA
>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
>>> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties
>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
>>> yarn             1073   0,1  1,0  3347132 174644 s001  S    10:29am
>>> 0:04.77
>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>>> -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
>>> -Dyarn.root.logger=INFO,RFA
>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
>>> -Dyarn.policy.file=hadoop-policy.xml -server
>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
>>> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
>>> -Dyarn.root.logger=INFO,RFA
>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
>>> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>
>>>  So I run it as yarn user in this way:
>>>
>>>  sudo -u yarn ….
>>>
>>>  Moreover I have just notice that I added the following lines into the
>>> hadoop-env.sh
>>>
>>>  hadoop-env.sh:export
>>> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>>
>>>  Accordingly to this other thread:
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C689.5060101@web.de%3E
>>>
>>>  Let me know,
>>> Alessandro
>>>
>>>
>>>  Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadamszx@gmail.com> ha
>>> scritto:
>>>
>>>  --f46d044283ecf837440508522939
>>> Content-Type: text/plain; charset=UTF-8
>>> Content-Transfer-Encoding: quoted-printable
>>>
>>> Ok, in the following, I use bin/yarn. Doesn't seem like much has changed:
>>>
>>> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh
>>> Running hduser script ...
>>> bin/yarn jar
>>>
>>> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for=
>>> -hadoop-2.5.1-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/hduser/input/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/hduser/output/shortestpaths -w 1 -yj
>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>> 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop
>>> library for your platform... using builtin-java classes where applicable
>>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format
>>> specified. Ensure your OutputFormat does not require one.
>>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is:
>>> hdfs://localhost:9000/user/hduser/output/shortestpaths
>>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client
>>> 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at
>>> localhost/127.0.0.1:8050
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM
>>> for,
>>> nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName
>>> /default-rack, nodeNumContainers 0
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>> size
>>> to 1024MB.
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application
>>> ID:
>>> application_1416310496070_0001
>>> 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is
>>> deprecated. Instead, use mapreduce.job.id
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the
>>> application master
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM
>>>
>>> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h=
>>>
>>> adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H=
>>>
>>> OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y=
>>>
>>> ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD=
>>>
>>> OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m=
>>> apreduce/lib/*}
>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources
>>> ::
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>> 496070_0001/giraph-conf.xml
>>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS
>>>
>>> :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name .
>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking .
>>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for
>>>
>>> :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja=
>>> r-with-dependencies.jar
>>> to
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>
>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>> encies.jar
>>> 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources
>>> ::
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>
>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>> encies.jar
>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient:
>>> ApplicationSumbissionContext
>>> for GiraphApplicationMaster launch container is populated.
>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to
>>> ASM
>>> 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application
>>> application_1416310496070_0001
>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after
>>> submission :application_1416310496070_0001
>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>> container request was submitted to ResourceManager for job: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01
>>> secs
>>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient:
>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
>>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86
>>> secs
>>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient:
>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
>>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02
>>> secs
>>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient:
>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
>>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39
>>> secs
>>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient:
>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
>>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57
>>> secs
>>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient:
>>> appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2
>>>
>>> gam_stderr.log
>>>
>>> LF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio=
>>>
>>> n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop=
>>>
>>> -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar=
>>> !/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> explana=
>>> tion.
>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>
>>> gam_stdout.log
>>>
>>> 2014-11-18 05:36:28,143 INFO  [main] yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
>>> 2014-11-18 05:36:30,404 WARN  [main] util.NativeCodeLoader
>>> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
>>> library for your platform... using builtin-java classes where
>>> applicable
>>> 2014-11-18 05:36:32,649 INFO  [main] yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId
>>> container_1416310496070_0001_01_000001 ApplicationAttemptId
>>> appattempt_1416310496070_0001_000001
>>> 2014-11-18 05:36:32,852 INFO  [main] client.RMProxy
>>> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
>>> localhost/127.0.0.1:8030
>>> 2014-11-18 05:36:32,888 INFO  [main] impl.NMClientAsyncImpl
>>> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread
>>> pool size is 500
>>> 2014-11-18 05:36:32,889 INFO  [main]
>>> impl.ContainerManagementProtocolProxy
>>> (ContainerManagementProtocolProxy.java:<init>(78)) -
>>> yarn.client.max-nodemanagers-proxies : 500
>>> 2014-11-18 05:36:33,589 INFO  [main] yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>>> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
>>> 2014-11-18 05:36:33,617 INFO  [main] yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>>> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
>>> 2014-11-18 05:36:33,617 INFO  [main] yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
>>> 2014-11-18 05:36:35,674 INFO  [AMRM Heartbeater thread]
>>> impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) -
>>> Received new token for : roosevelt:60486
>>> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
>>> response from RM for container ask, allocatedCnt=3D1
>>> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
>>> allocated # of container so far : 1 allocated out of 2 required.
>>> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>>> Launching command on a new container.,
>>> containerId=3Dcontainer_1416310496070_0001_01_000002,
>>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
>>> containerResourceMemory=3D1024
>>> 2014-11-18 05:36:35,688 INFO  [pool-4-thread-1]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
>>> Setting up container launch container for
>>> containerid=3Dcontainer_1416310496070_0001_01_000002
>>> 2014-11-18 05:36:35,720 INFO  [pool-4-thread-1]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
>>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
>>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1
>>> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
>>> 2014-11-18 05:36:35,721 INFO  [pool-4-thread-1]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
>>> Setting username in ContainerLaunchContext to: hduser
>>> 2014-11-18 05:36:36,789 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
>>> response from RM for container ask, allocatedCnt=3D1
>>> 2014-11-18 05:36:36,789 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
>>> allocated # of container so far : 2 allocated out of 2 required.
>>> 2014-11-18 05:36:36,790 INFO  [AMRM Callback Handler Thread]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>>> Launching command on a new container.,
>>> containerId=3Dcontainer_1416310496070_0001_01_000003,
>>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
>>> containerResourceMemory=3D1024
>>> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
>>> Setting up container launch container for
>>> containerid=3Dcontainer_1416310496070_0001_01_000003
>>> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
>>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
>>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1
>>> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
>>> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
>>> yarn.GiraphApplicationMaster
>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
>>> Setting username in ContainerLaunchContext to: hduser
>>> 2014-11-18 05:36:37,463 INFO  [pool-4-thread-1] yarn.YarnUtils
>>> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>> to LocalResources for export.to
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>
>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>> encies.jar
>>> 2014-11-18 05:36:37,678 INFO  [pool-4-thread-1] yarn.YarnUtils
>>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
>>> LocalResources ::
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>
>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>> encies.jar
>>> 2014-11-18 05:36:37,678 WARN  [pool-4-thread-1] yarn.YarnUtils
>>> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
>>> didn't include giraph-core.
>>> 2014-11-18 05:36:37,680 INFO  [pool-4-thread-1] yarn.YarnUtils
>>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
>>> LocalResources ::
>>>
>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>> 496070_0001/giraph-conf.xml
>>> 2014-11-18 05:36:37,694 INFO
>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>> Event EventType: START_CONTAINER for Container
>>> container_1416310496070_0001_01_000002
>>> 2014-11-18 05:36:37,699 INFO
>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>> Event EventType: START_CONTAINER for Container
>>> container_1416310496070_0001_01_000003
>>> 2014-11-18 05:36:37,698 INFO
>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>>> impl.ContainerManagementProtocolProxy
>>> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy
>>> : roosevelt:60486
>>> 2014-11-18 05:36:37,761 INFO
>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>> Event EventType: QUERY_CONTAINER for Container
>>> container_1416310496070_0001_01_000003
>>> 2014-11-18 05:36:37,771 INFO
>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>> Event EventType: QUERY_CONTAINER for Container
>>> container_1416310496070_0001_01_000002
>>>
>>>
>>>
>>> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegro81@yahoo.it>
>>> wrote:
>>>
>>> Hi Das,
>>> I mean as user you run Hadoop yarn resource manager.
>>>
>>> Alessandro
>>>
>>> Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadamszx@gmail.com> ha
>>> scritto:
>>>
>>> Alessandro,
>>>     I'm not sure what you mean, should I create a new user on my system
>>> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure.
>>>
>>> Tripti,
>>>     I'll get those logs as soon as I can.
>>>
>>> Thank you both,
>>>
>>> V/r
>>> Das
>>> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>>>
>>> Hi Tripti,
>>> I agree that a more detailed error log could be useful.
>>>
>>> Thanks,
>>> Alessandro
>>>
>>>
>>>
>>>
>>> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tripti@yahoo-inc.co
>>> =
>>>
>>>  m>
>>>
>>> ha scritto:
>>>
>>> --_000_D092AB9C1CB42triptiyahooinccom_
>>> Content-Type: text/plain; charset=3D"Windows-1252"
>>> Content-Transfer-Encoding: quoted-printable
>>>
>>> Hi Alessandro,
>>>
>>> I guess u r talking about the documentation on Giraph webpage which
>>>
>>> mention=3D
>>>
>>> s addition of this new option which is more or less mandatory for yarn
>>>
>>> base=3D
>>>
>>> d profiles.
>>> When I first ran the Giraph on yarn without the proper =3D96yj option,
>>>
>>> there =3D
>>>
>>> was no way I could figure out from the Application/Container logs that
>>>
>>> this=3D
>>>
>>> was the issue.
>>> I think, it=3D92ll be a good idea to have this message in the logs for
>>>
>>> easy d=3D
>>>
>>> ebugging.
>>>
>>> Thanks,
>>> Tripti.
>>>
>>>
>>> From: Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it
>>> <alenegro81@yahoo.it>
>>>
>>>
>>>  Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org
>>> <user@giraph.apache.org>>"
>>>
>>> <user@gir=3D
>>>
>>> aph.apache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
>>> Date: Tuesday, November 18, 2014 at 4:31 PM
>>> To: "user@giraph.apache.org<mailto:user@giraph.apache.org
>>> <user@giraph.apache.org>>" <
>>>
>>> user@giraph.ap=3D
>>>
>>> ache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
>>> Subject: Re: Help with Giraph on Yarn
>>>
>>> Hi Eli,
>>> no I think that the help message is clear enough, generally when I rea=
>>>
>>>  d
>>>
>>> =3D93=3D
>>>
>>> jar=3D94 I always mean absolute path for jar, but this is a my persona=
>>>
>>>  l
>>>
>>> misun=3D
>>>
>>> derstanding.
>>>
>>> Thanks,
>>> Alessandro
>>>
>>>
>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <
>>>
>>> apache.mailbox@gmail.co=3D
>>>
>>> m<mailto:apache.mailbox@gmail.com <apache.mailbox@gmail.com>>> ha
>>> scritto:
>>>
>>> Sounds like you got it figured out since last time I checked this list=
>>>
>>>  ,
>>>
>>> sor=3D
>>>
>>> ry it was a pain. Feel free to drop a JIRA ticket if we can make help
>>>
>>> messa=3D
>>>
>>> ge etc. for -yj option more clear, there's lots to improve there.
>>>
>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegro81@yahoo.it
>>>
>>> <mailt=3D
>>>
>>> o:alenegro81@yahoo.it>> wrote:
>>> Hi Tripti,
>>> finally I was able to run the test with success. It was an issue of
>>>
>>> permiss=3D
>>>
>>> ion since I was running as ale not as yarn.
>>> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1.
>>>
>>> This is=3D
>>>
>>> the final result:
>>>
>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>
>>> org.apache.=3D
>>>
>>> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total
>>>
>>> running ti=3D
>>>
>>> me: 0 minutes, 21 seconds.
>>>
>>>
>>> Many thanks for your support,
>>> Alessandro
>>>
>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <
>>>
>>> tripti@yahoo-inc.com<m=3D
>>>
>>> ailto:tripti@yahoo-inc.com>> ha scritto:
>>>
>>> I don't know if u have access to this node. But if u do, u could check
>>>
>>> if t=3D
>>>
>>> he file is indeed there and u have access to it.
>>>
>>> Sent from my iPhone
>>>
>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegro81@yahoo.it
>>>
>>> <mailto:=3D <=3D>
>>>
>>> alenegro81@yahoo.it>> wrote:
>>>
>>> You are right it works, but now I receive the following error:
>>>
>>> SLF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>
>>>
>>>
>>> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e=
>>>
>>>  xam=3D
>>>
>>>
>>>
>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>>>
>>>  j/i=3D
>>>
>>>  mpl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>>>
>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>>>
>>>  ]
>>>
>>>  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>
>>> explana=3D
>>>
>>> tion.
>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro=
>>>
>>>  m
>>>
>>> SCD=3D
>>>
>>> ynamicStore
>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException:
>>>
>>> Coul=3D
>>>
>>> d not configure the containerlaunch context for GiraphYarnTasks.
>>> at
>>>
>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>>>
>>>  =3D
>>>
>>>  ApplicationMaster.java:391)
>>> at
>>>
>>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat=
>>>
>>>  =3D
>>>
>>>  ionMaster.java:78)
>>> at
>>>
>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b=
>>>
>>>  =3D
>>>
>>>  uildContainerLaunchContext(GiraphApplicationMaster.java:522)
>>> at
>>>
>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r=
>>>
>>>  =3D
>>>
>>>  un(GiraphApplicationMaster.java:479)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
>>>
>>>  =3D
>>>
>>>  a:1145)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
>>>
>>>  =3D
>>>
>>>  va:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: java.io.FileNotFoundException: File does not exist:
>>>
>>> hdfs://hadoo=3D
>>>
>>>
>>>
>>> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_=
>>>
>>>  000=3D
>>>
>>>
>>>
>>> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.=
>>>
>>>  jar
>>>
>>>  at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>>>
>>>  =3D
>>>
>>>  stem.java:1072)
>>> at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>>>
>>>  =3D
>>>
>>>  stem.java:1064)
>>> at
>>>
>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv=
>>>
>>>  =3D
>>>
>>>  er.java:81)
>>> at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi=
>>>
>>>  =3D
>>>
>>>  leSystem.java:1064)
>>> at
>>>
>>> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153=
>>>
>>>  =3D
>>>
>>>  )
>>> at
>>>
>>> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
>>>
>>> at
>>>
>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>>>
>>>  =3D
>>>
>>>  ApplicationMaster.java:387)
>>> ... 6 more
>>>
>>>
>>> That justify the other error I receive in the task:
>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>>>
>>>  k
>>>
>>>
>>> Thanks,
>>>
>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <
>>>
>>> tripti@yahoo-inc.com<m=3D
>>>
>>> ailto:tripti@yahoo-inc.com>> ha scritto:
>>>
>>> Why r u adding two jars? Example jar ideally contains core library so
>>>
>>> every=3D
>>>
>>> thing should be available with just one example jar included
>>>
>>> Sent from my iPhone
>>>
>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it
>>>
>>> <mailto:=3D <=3D>
>>>
>>> alenegro81@yahoo.it>> wrote:
>>>
>>> Hi,
>>> now it seems better, I need to add:
>>>
>>>
>>>
>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-=
>>>
>>>  exa=3D
>>>
>>>  mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>
>>>
>>> Now it seems that after a lot of cycle it fail with this error:
>>>
>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>>>
>>>  k
>>>
>>>
>>> But in this case the error appear in task-3-stderr.log  not in
>>>
>>> gam-stderr.l=3D
>>>
>>> og where there is the following error:
>>>
>>> LF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>
>>>
>>>
>>> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1=
>>>
>>>  .1.=3D
>>>
>>>
>>>
>>> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St=
>>>
>>>  ati=3D
>>>
>>>  cLoggerBinder.class]
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>
>>>
>>>
>>> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e=
>>>
>>>  xam=3D
>>>
>>>
>>>
>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>>>
>>>  j/i=3D
>>>
>>>  mpl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>>
>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>>>
>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>>>
>>>  ]
>>>
>>>  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>
>>>
>>>
>>>
>>
>
>

Mime
View raw message