giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tripti Singh <tri...@yahoo-inc.com>
Subject Re: Help with Giraph on Yarn
Date Thu, 06 Nov 2014 12:07:31 GMT
Why r u adding two jars? Example jar ideally contains core library so everything should be
available with just one example jar included

Sent from my iPhone

On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:

Hi,
now it seems better, I need to add:

giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar


Now it seems that after a lot of cycle it fail with this error:

Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask

But in this case the error appear in task-3-stderr.log  not in gam-stderr.log where there
is the following error:

LF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDynamicStore
Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure
the containerlaunch context for GiraphYarnTasks.
at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391)
at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78)
at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522)
at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0006/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153)
at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387)
... 6 more


Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tripti@yahoo-inc.com<mailto:tripti@yahoo-inc.com>>
ha scritto:

Hi,
I remember seeing this thread, but since i was already deep into fixing this issue in production,
I didn't really wait for the patch, not even the munge symbol as I was not even able to compile
the yarn profile( sorry for being so ruthless to giraph :-( )
I did later cross check with the patch and it was quite similar to my change but sure it wouldn't
work for all profiles(in fact some profiles may even break due to Hadoop class incompatibility
) I can share those changes with u once I get back to my computer but I don't think that could
have caused this problem.
Just try the small changed I mentioned in my previous reply and let me know if that helped.

Thanks,
Tripti
Sent from my iPhone

On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:

Hi Tripti,
let me add one info, As you can see from this thread of email:

http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3CC3F75A98-BDE8-4205-971F-1F7241E6E83F@yahoo.it%3E

I had some issue while compiling for Hadoop 2.5.1 could be that into the process I loose something?

Thanks again,
Alessandro



Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
ha scritto:

Hi Tripti,
yes I run in this way:


$HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1 -yj
/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar

And this is the complete trace:

2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from SCDynamicStore
14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your
InputFormat does not require one.
14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your
OutputFormat does not require one.
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client
14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=10.61.137.28:59677,
nodeAddress 10.61.137.28:8042, nodeRackName /default-rack, nodeNumContainers 0
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415264041937_0005
14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use
mapreduce.job.id
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the application master
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name .
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking .
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster
launch container is populated.
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was
submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 0,83 secs
14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005_000001, State:
ACCEPTED, Containers used: 1
14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
reports FAILED state, diagnostics show: Application application_1415264041937_0005 failed
2 times due to AM Container for appattempt_1415264041937_0005_000002 exited with  exitCode:
1 due to: Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory
for Giraph job.
14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation:
FAILED, total running time: 0 minutes, 3 seconds.
ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_1415264041937_0005

As you can see there is this piece:

14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar


But I continue to receive the error:

Thanks,
Alessandro


Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tripti@yahoo-inc.com<mailto:tripti@yahoo-inc.com>>
ha scritto:

For GiraphApplicationMaster not found, the simple thing that worked for me was adding a -yj
parameter to my command.
(hadoop jar {jar-file-path} {path to giraph runner} {computation-class} {bunch of parameters}
-yj {jar-file-name})
If u look at job logs,
Do u see a log statement defining "LIB JARS"?

Sent from my iPhone

On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:

Hi,
yea my logs shows that error. I follow this example:

http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C689.5060101@web.de%3E

without success.
Then I try to add all jars I can from giraph adding the following lines into the hadoop-env.sh:

export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB


when I run the application now I see this:

14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib


But again the same error. What I think is that there is no problem in the library but while
is starting up.
I saw this piece in the logs:

14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster
launch container is populated.
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was
submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 0,83 secs


That show that it is able to load GiraphApplicationMaster but fail during the startup.
Where can I find more logs?

Thanks,
Alessandro


Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tripti@yahoo-inc.com<mailto:tripti@yahoo-inc.com>>
ha scritto:

Can u run the following command to find container logs and post it here:

yarn logs -applicationId application_1415203231170_0002

If u find this error:

Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster

It may just be due to non-availablity of the yarn jars

Thanks,
Tripti
Tech Yahoo

From: Jing Fan <fanjing09@gmail.com<mailto:fanjing09@gmail.com>>
Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>>
Date: Thursday, November 6, 2014 at 1:56 AM
To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>>
Subject: Re: Help with Giraph on Yarn

Then do you know to solve this error?

14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed
2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with  exitCode:
1 due to: Exception from container-launch.
Container id: container_1415203231170_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:
Hi,
in the mean time I found the reason why it hangs for long time here:

base path checking .


It scan all the directory tree starting from current directory, if you run it into your home
(and you have a lot of files and directories) it takes ages to complete.

I continue to got the same error.

Regards,
Alessandro

Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjing09@gmail.com<mailto:fanjing09@gmail.com>>
ha scritto:

I run with:

hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/jing/shortestpaths
-w 1 -yj giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar

On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:
This is my complete logs with the command I run:


ale:~ />$HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1
2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from SCDynamicStore
14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your
InputFormat does not require one.
14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your
OutputFormat does not require one.
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client
14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032/>
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=192.168.0.105:50658<http://192.168.0.105:50658/>,
nodeAddress 192.168.0.105:8042<http://192.168.0.105:8042/>, nodeRackName /default-rack,
nodeNumContainers 0
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415189069846_0004
14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id<http://mapred.job.id/>
is deprecated. Instead, use mapreduce.job.id<http://mapreduce.job.id/>
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the application master
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml
14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS :
14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name .
14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking .
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster
launch container is populated.
14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application application_1415189069846_0004
14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415189069846_0004
14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was
submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 0,82 secs
14/11/05 14:42:57 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004_000001, State:
ACCEPTED, Containers used: 1
14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
reports FAILED state, diagnostics show: Application application_1415189069846_0004 failed
2 times due to AM Container for appattempt_1415189069846_0004_000002 exited with  exitCode:
1 due to: Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory
for Giraph job.
14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation:
FAILED, total running time: 0 minutes, 3 seconds.


Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <tripti@yahoo-inc.com<mailto:tripti@yahoo-inc.com>>
ha scritto:

I faced the similar problem where the workflow just hangs.
Can u see in ur logs if there is a task with id 0.
I made a local fix and I could give u the patch if u see a similar problem and ur workflow
is unable to find the zookeeper manager file and repeats the following message:
For task xyz, got file 'null'

Thanks,
Tripti
Sent from my iPhone

On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:

Hi,
have you seen the hadoop logs?
Could you write here how are you running the command?

Thanks,
Alessandro

Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjing09@gmail.com<mailto:fanjing09@gmail.com>>
ha scritto:

Yeah, I add -yj option. The result is the same. Still error here.

On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
wrote:
Hi Jing,
I'm facing the same problem. Do you check the hadoop logs? In my case I got:

Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster


Moreover I don't know why my application hang for a long time here:

14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking .

And then continue failing with your error into the output and the other into the hadoop logs.

Did you test adding this?

-yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar

Thanks,
Alessandro


Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjing09@gmail.com<mailto:fanjing09@gmail.com>>
ha scritto:

Hi,

I install the Giraph with Yarn.
But when I run the program with following command:

 hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/j/tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/j/shortestpaths -w
1

The program exited with error:

14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 0.83 secs
14/11/05 10:25:14 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State:
ACCEPTED, Containers used: 0
14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 4.84 secs
14/11/05 10:25:18 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State:
ACCEPTED, Containers used: 0
14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 8.85 secs
14/11/05 10:25:22 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State:
ACCEPTED, Containers used: 0
14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 12.86 secs
14/11/05 10:25:26 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State:
ACCEPTED, Containers used: 0
14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed
2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with  exitCode:
1 due to: Exception from container-launch.
Container id: container_1415203231170_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


Anyone can help?













Mime
View raw message