giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From D Adams <dadam...@gmail.com>
Subject Re: Help with Giraph on Yarn
Date Sat, 22 Nov 2014 17:42:53 GMT
I'm sorry, please excuse my ignorance, but, I have no idea where to find my
configuration. That is, I have no idea where these lines came from (or what
file):

yarn             1029   0,1  1,2  3499572 209560 s001  S    10:29am
0:06.04 /Library/Java/
JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
-Dproc_resourcemanager -Xmx500m
-Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
-Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
...
As far as configuration goes, I'm only really aware of the
etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above
part of the command you use to run giraph jobs?
I will make changes to hadoop-env.sh file as you suggest and let you know
how it works out.

On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro <alenegro81@yahoo.it>
wrote:

> Hi,
> I forgot to say that I added also this line:
>
> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>
>
> Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro <
> alenegro81@yahoo.it> ha scritto:
>
> Hi Das,
> this is my configuration of hadoop:
>
> yarn             1029   0,1  1,2  3499572 209560 s001  S    10:29am
> 0:06.04
> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
> -Dproc_resourcemanager -Xmx500m
> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
> -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
> -Dyarn.policy.file=hadoop-policy.xml
> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
> -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> yarn             1073   0,1  1,0  3347132 174644 s001  S    10:29am
> 0:04.77
> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
> -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
> -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
> -Dyarn.policy.file=hadoop-policy.xml -server
> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
> -Dyarn.root.logger=INFO,RFA
> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>
> So I run it as yarn user in this way:
>
> sudo -u yarn ….
>
> Moreover I have just notice that I added the following lines into the
> hadoop-env.sh
>
> hadoop-env.sh:export
> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
>
> Accordingly to this other thread:
>
>
> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C689.5060101@web.de%3E
>
> Let me know,
> Alessandro
>
>
> Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadamszx@gmail.com> ha
> scritto:
>
> --f46d044283ecf837440508522939
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
>
> Ok, in the following, I use bin/yarn. Doesn't seem like much has changed:
>
> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh
> Running hduser script ...
> bin/yarn jar
>
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for=
> -hadoop-2.5.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/hduser/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/output/shortestpaths -w 1 -yj
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://localhost:9000/user/hduser/output/shortestpaths
> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client
> 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8050
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM for,
> nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName
> /default-rack, nodeNumContainers 0
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
> to 1024MB.
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1416310496070_0001
> 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM
>
> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h=
>
> adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H=
>
> OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y=
>
> ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD=
>
> OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m=
> apreduce/lib/*}
> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources ::
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
> 496070_0001/giraph-conf.xml
> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS
> :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name .
> 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking .
> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for
>
> :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja=
> r-with-dependencies.jar
> to
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources ::
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
> for GiraphApplicationMaster launch container is populated.
> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application
> application_1416310496070_0001
> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after
> submission :application_1416310496070_0001
> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01
> secs
> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient:
> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86
> secs
> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient:
> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02
> secs
> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient:
> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39
> secs
> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient:
> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1
> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57
> secs
> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient:
> appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2
>
> gam_stderr.log
>
> LF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio=
>
> n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop=
> -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar=
> !/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explana=
> tion.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> gam_stdout.log
>
> 2014-11-18 05:36:28,143 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
> 2014-11-18 05:36:30,404 WARN  [main] util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
> library for your platform... using builtin-java classes where
> applicable
> 2014-11-18 05:36:32,649 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId
> container_1416310496070_0001_01_000001 ApplicationAttemptId
> appattempt_1416310496070_0001_000001
> 2014-11-18 05:36:32,852 INFO  [main] client.RMProxy
> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
> localhost/127.0.0.1:8030
> 2014-11-18 05:36:32,888 INFO  [main] impl.NMClientAsyncImpl
> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread
> pool size is 500
> 2014-11-18 05:36:32,889 INFO  [main]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:<init>(78)) -
> yarn.client.max-nodemanagers-proxies : 500
> 2014-11-18 05:36:33,589 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
> 2014-11-18 05:36:33,617 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
> 2014-11-18 05:36:33,617 INFO  [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
> 2014-11-18 05:36:35,674 INFO  [AMRM Heartbeater thread]
> impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) -
> Received new token for : roosevelt:60486
> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
> response from RM for container ask, allocatedCnt=3D1
> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
> allocated # of container so far : 1 allocated out of 2 required.
> 2014-11-18 05:36:35,681 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
> Launching command on a new container.,
> containerId=3Dcontainer_1416310496070_0001_01_000002,
> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
> containerResourceMemory=3D1024
> 2014-11-18 05:36:35,688 INFO  [pool-4-thread-1]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
> Setting up container launch container for
> containerid=3Dcontainer_1416310496070_0001_01_000002
> 2014-11-18 05:36:35,720 INFO  [pool-4-thread-1]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1
> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
> 2014-11-18 05:36:35,721 INFO  [pool-4-thread-1]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
> Setting username in ContainerLaunchContext to: hduser
> 2014-11-18 05:36:36,789 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
> response from RM for container ask, allocatedCnt=3D1
> 2014-11-18 05:36:36,789 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
> allocated # of container so far : 2 allocated out of 2 required.
> 2014-11-18 05:36:36,790 INFO  [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
> Launching command on a new container.,
> containerId=3Dcontainer_1416310496070_0001_01_000003,
> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
> containerResourceMemory=3D1024
> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
> Setting up container launch container for
> containerid=3Dcontainer_1416310496070_0001_01_000003
> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1
> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
> 2014-11-18 05:36:37,227 INFO  [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
> Setting username in ContainerLaunchContext to: hduser
> 2014-11-18 05:36:37,463 INFO  [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> to LocalResources for export.to
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 2014-11-18 05:36:37,678 INFO  [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 2014-11-18 05:36:37,678 WARN  [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
> didn't include giraph-core.
> 2014-11-18 05:36:37,680 INFO  [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
>
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
> 496070_0001/giraph-conf.xml
> 2014-11-18 05:36:37,694 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: START_CONTAINER for Container
> container_1416310496070_0001_01_000002
> 2014-11-18 05:36:37,699 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: START_CONTAINER for Container
> container_1416310496070_0001_01_000003
> 2014-11-18 05:36:37,698 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy
> : roosevelt:60486
> 2014-11-18 05:36:37,761 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: QUERY_CONTAINER for Container
> container_1416310496070_0001_01_000003
> 2014-11-18 05:36:37,771 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: QUERY_CONTAINER for Container
> container_1416310496070_0001_01_000002
>
>
>
> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegro81@yahoo.it>
> wrote:
>
> Hi Das,
> I mean as user you run Hadoop yarn resource manager.
>
> Alessandro
>
> Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadamszx@gmail.com> ha
> scritto:
>
> Alessandro,
>     I'm not sure what you mean, should I create a new user on my system
> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure.
>
> Tripti,
>     I'll get those logs as soon as I can.
>
> Thank you both,
>
> V/r
> Das
> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
>
> Hi Tripti,
> I agree that a more detailed error log could be useful.
>
> Thanks,
> Alessandro
>
>
>
>
> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tripti@yahoo-inc.co=
>
> m>
>
> ha scritto:
>
> --_000_D092AB9C1CB42triptiyahooinccom_
> Content-Type: text/plain; charset=3D"Windows-1252"
> Content-Transfer-Encoding: quoted-printable
>
> Hi Alessandro,
>
> I guess u r talking about the documentation on Giraph webpage which
>
> mention=3D
>
> s addition of this new option which is more or less mandatory for yarn
>
> base=3D
>
> d profiles.
> When I first ran the Giraph on yarn without the proper =3D96yj option,
>
> there =3D
>
> was no way I could figure out from the Application/Container logs that
>
> this=3D
>
> was the issue.
> I think, it=3D92ll be a good idea to have this message in the logs for
>
> easy d=3D
>
> ebugging.
>
> Thanks,
> Tripti.
>
>
> From: Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it
> <alenegro81@yahoo.it>
>
>
> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org
> <user@giraph.apache.org>>"
>
> <user@gir=3D
>
> aph.apache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
> Date: Tuesday, November 18, 2014 at 4:31 PM
> To: "user@giraph.apache.org<mailto:user@giraph.apache.org
> <user@giraph.apache.org>>" <
>
> user@giraph.ap=3D
>
> ache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
> Subject: Re: Help with Giraph on Yarn
>
> Hi Eli,
> no I think that the help message is clear enough, generally when I rea=
>
> d
>
> =3D93=3D
>
> jar=3D94 I always mean absolute path for jar, but this is a my persona=
>
> l
>
> misun=3D
>
> derstanding.
>
> Thanks,
> Alessandro
>
>
> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <
>
> apache.mailbox@gmail.co=3D
>
> m<mailto:apache.mailbox@gmail.com <apache.mailbox@gmail.com>>> ha scritto:
>
> Sounds like you got it figured out since last time I checked this list=
>
> ,
>
> sor=3D
>
> ry it was a pain. Feel free to drop a JIRA ticket if we can make help
>
> messa=3D
>
> ge etc. for -yj option more clear, there's lots to improve there.
>
> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegro81@yahoo.it
>
> <mailt=3D
>
> o:alenegro81@yahoo.it>> wrote:
> Hi Tripti,
> finally I was able to run the test with success. It was an issue of
>
> permiss=3D
>
> ion since I was running as ale not as yarn.
> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1.
>
> This is=3D
>
> the final result:
>
> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph:
>
> org.apache.=3D
>
> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total
>
> running ti=3D
>
> me: 0 minutes, 21 seconds.
>
>
> Many thanks for your support,
> Alessandro
>
> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <
>
> tripti@yahoo-inc.com<m=3D
>
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> I don't know if u have access to this node. But if u do, u could check
>
> if t=3D
>
> he file is indeed there and u have access to it.
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegro81@yahoo.it
>
> <mailto:=3D
>
> alenegro81@yahoo.it>> wrote:
>
> You are right it works, but now I receive the following error:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>
>
> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e=
>
> xam=3D
>
>
> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>
> j/i=3D
>
> mpl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>
> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>
> ]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>
> explana=3D
>
> tion.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro=
>
> m
>
> SCD=3D
>
> ynamicStore
> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException:
>
> Coul=3D
>
> d not configure the containerlaunch context for GiraphYarnTasks.
> at
>
> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>
> =3D
>
> ApplicationMaster.java:391)
> at
>
> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat=
>
> =3D
>
> ionMaster.java:78)
> at
>
> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b=
>
> =3D
>
> uildContainerLaunchContext(GiraphApplicationMaster.java:522)
> at
>
> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r=
>
> =3D
>
> un(GiraphApplicationMaster.java:479)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
>
> =3D
>
> a:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
>
> =3D
>
> va:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException: File does not exist:
>
> hdfs://hadoo=3D
>
>
> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_=
>
> 000=3D
>
>
> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.=
>
> jar
>
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>
> =3D
>
> stem.java:1072)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>
> =3D
>
> stem.java:1064)
> at
>
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv=
>
> =3D
>
> er.java:81)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi=
>
> =3D
>
> leSystem.java:1064)
> at
>
> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153=
>
> =3D
>
> )
> at
>
> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
>
> at
>
> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>
> =3D
>
> ApplicationMaster.java:387)
> ... 6 more
>
>
> That justify the other error I receive in the task:
> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>
> k
>
>
> Thanks,
>
> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <
>
> tripti@yahoo-inc.com<m=3D
>
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> Why r u adding two jars? Example jar ideally contains core library so
>
> every=3D
>
> thing should be available with just one example jar included
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it
>
> <mailto:=3D
>
> alenegro81@yahoo.it>> wrote:
>
> Hi,
> now it seems better, I need to add:
>
>
> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-=
>
> exa=3D
>
> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
>
> Now it seems that after a lot of cycle it fail with this error:
>
> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>
> k
>
>
> But in this case the error appear in task-3-stderr.log  not in
>
> gam-stderr.l=3D
>
> og where there is the following error:
>
> LF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>
>
> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1=
>
> .1.=3D
>
>
> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St=
>
> ati=3D
>
> cLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>
>
> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e=
>
> xam=3D
>
>
> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>
> j/i=3D
>
> mpl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>
> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>
> ]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>
>
>
>

Mime
View raw message