giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Sirotin <sirot...@web.de>
Subject GiraphApplicationMaster not found Giraph 1.1.0 for Hadoop 2.4 Jdk 1.7 Ubuntu
Date Sat, 26 Jul 2014 13:15:37 GMT
Hello together,

I am trying to run the Example-Jar from Giraph since at least 2 days.

Following I have tried:

1.

  *

    git clone https://git-wip-us.apache.org/repos/asf/giraph.git

  *

    mv giraph giraph_src; cd giraph_src

  *

    mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests clean package

  *

    echo export PATH=\$PATH:/usr/local/giraph/bin >> ~/.bashrc

  *

    echo export GIRAPH_HOME=/usr/local/giraph >> ~/.bashrc

  *

    echo export HADOOP_MAPRED_HOME=$HADOOP_PREFIX >> ~/.bashrc

  *

    source ~/.bashrc

  *

    jar=$GIRAPH_HOME/giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar

  *

    runner=org.apache.giraph.GiraphRunner

  *

    computation=org.apache.giraph.examples.SimpleShortestPathsComputation

  *

    informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat

  *

    outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat


  *

    hadoop jar $jar $runner $computation -vif $informat -vip
    tiny_graph.txt -vof $outformat -op shortestpaths -w 1 -yj $jar


I have also put the Example-Jar to /usr/local/hadoop/share/hadoop and 
restarted the single-cluster. But still the last command aborts and in 
the userlogs you can see, that the main-class 
org.apache.giraph.yarn.GiraphApplicationMaster couldnt be found.

How can u tell the taskrunner where to find the jar else?

2. I ve done the same compiling Giraph with Hadoop-version 2.2 and got 
the same error.

3. Then I ve reverted the repository with git revert 
ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 described like on 
http://mail-archives.apache.org/mod_mbox/giraph-user/201312.mbox/%3C1647021.5fbjhLDxPK@chronos7%3E.
But the revert got conflicts on quite 5 pom.xml-files - so I gave up, 
while I am still mvn-newbie

4. Then I made a git fetch origin 
ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 described like on 
http://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset.
I got a clean state, but the mvn clean install|package went to errors 
because of version-conflicts of many packages of the Apache Giraph Parent


This is the following output after executing the last command of 1. and 2.:


hadoop jar $jar $runner $computation -vif $informat -vip tiny_graph.txt 
-vof $outformat -op shortestpaths -w 1 -yj $jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop-2.4.0/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop-2.4.0/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/07/26 13:15:45 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
14/07/26 13:15:46 INFO utils.ConfigurationUtils: No edge input format 
specified. Ensure your InputFormat does not require one.
14/07/26 13:15:46 INFO utils.ConfigurationUtils: No edge output format 
specified. Ensure your OutputFormat does not require one.
14/07/26 13:15:46 INFO yarn.GiraphYarnClient: Final output path is: 
hdfs://vmvis5:9000/user/hadoop/shortestpaths
14/07/26 13:15:46 INFO yarn.GiraphYarnClient: Running Client
14/07/26 13:15:46 INFO client.RMProxy: Connecting to ResourceManager at 
vmvis5/141.71.31.187:8032
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Got node report from ASM 
for, nodeId=vmvis5:48804, nodeAddress vmvis5:8042, nodeRackName 
/default-rack, nodeNumContainers 0
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Defaulting per-task heap 
size to 1024MB.
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Obtained new Application 
ID: application_1406373194228_0003
14/07/26 13:15:47 INFO Configuration.deprecation: mapred.job.id is 
deprecated. Instead, use mapreduce.job.id
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Set the environment for 
the application master
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Environment for AM 
:{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/07/26 13:15:47 INFO yarn.YarnUtils: Registered file in LocalResources 
:: 
hdfs://vmvis5:9000/user/hadoop/giraph_yarn_jar_cache/application_1406373194228_0003/giraph-conf.xml
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: LIB JARS 
:/usr/local/giraph/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name .
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking .
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name /usr/local/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking /usr/local/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/etc/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/etc/hadoop
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/common/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/common/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/common/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/common
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/hdfs/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/hdfs/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/hdfs/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/hdfs
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/yarn/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/yarn/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/yarn/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/yarn
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/mapreduce/lib/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/mapreduce/lib
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/hadoop/mapreduce/*
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/hadoop/mapreduce
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/contrib/capacity-scheduler/*.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/contrib/capacity-scheduler
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/myLib/giraph-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: Class path name 
/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.YarnUtils: base path checking 
/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.2.0-jar-with-dependencies.jar
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: 
ApplicationSumbissionContext for GiraphApplicationMaster launch 
container is populated.
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/07/26 13:15:47 INFO impl.YarnClientImpl: Submitted application 
application_1406373194228_0003
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: Got new appId after 
submission :application_1406373194228_0003
14/07/26 13:15:47 INFO yarn.GiraphYarnClient: GiraphApplicationMaster 
container request was submitted to ResourceManager for job: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation
14/07/26 13:15:48 INFO yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,85 
secs
14/07/26 13:15:48 INFO yarn.GiraphYarnClient: 
appattempt_1406373194228_0003_000001, State: ACCEPTED, Containers used: 1
14/07/26 13:15:51 ERROR yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED 
state, diagnostics show: Application application_1406373194228_0003 
failed 2 times due to AM Container for 
appattempt_1406373194228_0003_000002 exited with exitCode: 1 due to: 
Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
         at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
         at org.apache.hadoop.util.Shell.run(Shell.java:418)
         at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
         at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
         at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
         at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/07/26 13:15:51 INFO yarn.GiraphYarnClient: Cleaning up HDFS 
distributed cache directory for Giraph job.
14/07/26 13:15:51 INFO yarn.GiraphYarnClient: Completed Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total 
running time: 0 minutes, 3 seconds.

<< END

I hope someone knows, what I did wrong. Maybe the Distribution-Cache in 
Yarn is working differently and I have todo first something?

Thank you for your help!

Sincerely
Alex



Mime
View raw message