giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Sirotin <sirot...@web.de>
Subject Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Date Tue, 12 Aug 2014 06:59:38 GMT
Hi,

you can increase the Heapsize to 2048 Mb by setting it into 
etc/hadoop/hadoop-env.sh. Search for "export HADOOP_HEAPSIZE="

But if I fix this, the TaskRunner and GiraphApplicationMaster will not 
stop to end. This is what I am seeing from the Client everytime repeating:

14/08/12 08:51:06 INFO yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 
1494,96 secs
14/08/12 08:51:06 INFO yarn.GiraphYarnClient: 
appattempt_1407762054368_0017_000001, State: RUNNING, Containers used: 2

The gam-stderr.log (GiraphApplicationMaster) showed the following:

aphYarnTask 1407762054368 17 2 1
         |- 13562 11012 13562 13562 (bash) 0 0 18870272 435 /bin/bash -c 
java -Xmx1024M -Xms1024M -cp 
.:${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:./*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*

org.apache.giraph.yarn.GiraphYarnTask 1407762054368 17 2 1 
1>/usr/local/hadoop/logs/userlogs/application_1407762054368_0017/container_1407762054368_0017_01_000002/task-2-stdout.log

2>/usr/local/hadoop/logs/userlogs/application_1407762054368_0017/container_1407762054368_0017_01_000002/task-2-stderr.log

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

14/08/12 08:26:23 INFO yarn.GiraphApplicationMaster: After completion of 
one conatiner. current status is: completedCount :1 containersToLaunch 
:2 successfulCount :0 failedCount :1


And the GiraphYarnTask also repeats the following:

14/08/12 08:56:15 WARN zookeeper.ClientCnxn: Session 0x147c8e553a10001 
for server null, unexpected error, closing socket connection and 
attempting reconnect
java.net.ConnectException: Verbindungsaufbau abgelehnt
         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
         at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
14/08/12 08:56:16 INFO zookeeper.ClientCnxn: Opening socket connection 
to server blabla:22181. Will not attempt to authenticate using SASL 
(unknown error)
14/08/12 08:56:16 WARN zookeeper.ClientCnxn: Session 0x147c8e553a10001 
for server null, unexpected error, closing socket connection and 
attempting reconnect
java.net.ConnectException: Verbindungsaufbau abgelehnt
         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
         at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Do you have an idea, how I can fix this? With ZooKeeper, I am also newbie

Thx




On 12.08.2014 08:14, Vikalp Handa wrote:
> Thanks a lot... I have build successfully even without running mvn 
> clean :P
> Now while executing the example code I am facing a different problem 
> of Cluster Heap Size.
>
> *How can I update this cluster heap size?*
>
> *Command: *hadoop jar 
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar

> org.apache.giraph.GiraphRunner 
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
> /user/hadoop/input/tiny_graph.txt -vof 
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
> /user/hadoop/outputpaths -yh 2048 -w 1
>
> *Result:*
> *
> *
> 14/08/12 11:37:15 WARN util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes 
> where applicable
> 14/08/12 11:37:15 INFO utils.ConfigurationUtils: No edge input format 
> specified. Ensure your InputFormat does not require one.
> 14/08/12 11:37:15 INFO utils.ConfigurationUtils: No edge output format 
> specified. Ensure your OutputFormat does not require one.
> 14/08/12 11:37:15 INFO yarn.GiraphYarnClient: Final output path is: 
> hdfs://localhost:9000/user/hadoop/outputpaths
> 14/08/12 11:37:15 INFO yarn.GiraphYarnClient: Running Client
> 14/08/12 11:37:15 INFO client.RMProxy: Connecting to ResourceManager 
> at /0.0.0.0:8032 <http://0.0.0.0:8032>
> Exception in thread "main" java.lang.IllegalStateException: Giraph's 
> estimated cluster heap 4096MB ask is greater than the current 
> available cluster heap of 0MB. Aborting Job.
>         at 
> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>         at 
> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:124)
>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
> Regards,
> Vikalp Handa
>
>
> On Tue, Aug 12, 2014 at 11:18 AM, 이성일 <si503.lee@samsung.com 
> <mailto:si503.lee@samsung.com>> wrote:
>
>     You just run 'mvn clean' before that command :)
>
>     Best Regards, Seong-il
>     ------- Original Message -------
>     Sender : Vikalp Handa<handa.vikalp@gmail.com
>     <mailto:handa.vikalp@gmail.com>>
>     Date : 2014-08-12 14:35 (GMT+09:00)
>     Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
>
>     Thanks for your replies. Do I need to uninstall anything before
>     running the above mvn command ?
>
>     Regards,
>     Vikalp Handa
>
>     On Tue, Aug 12, 2014 at 10:26 AM, 이성일 <si503.lee@samsung.com
>     <mailto:si503.lee@samsung.com>> wrote:
>     Hi, Vikalp
>     I think you complied as wrong hadoop version.
>     Try below command
>     If you use yarn
>     mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package
>     or if you use just hadoop2
>     mvn -Phadoop_2  -Dhadoop.version=2.4.0 -DskipTests package
>     Best Regards, Seong-il
>
>     ------- Original Message -------
>     Sender : Vikalp Handa<handa.vikalp@gmail.com
>     <mailto:handa.vikalp@gmail.com>>
>     Date : 2014-08-12 13:40 (GMT+09:00)
>     Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x
>
>     Hi everyone,
>     I am new to Apache Giraph and would like execute Shortestpath and
>     PageRank example code on Hadoop 2.4.0 single node cluster (my
>     machine) having Centos 6.5.
>     I have successfully build Giraph on my machine but unable to
>     execute ShortestPath code.
>     Please let me know if there are any dependencies to be resolved
>     before code execution.
>
>     P.S. :
>     Command used:
>     hadoop jar
>     /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>     org.apache.giraph.GiraphRunner
>     org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>     org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>     -vip /user/hadoop/input/tiny_graph.txt -vof
>     org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>     /user/hduser/output/shortestpaths -w 1  -ca
>     giraph.SplitMasterWorker=false
>
>     Execution Result:
>     14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load
>     native-hadoop library for your platform... using builtin-java
>     classes where applicable
>     14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input
>     format specified. Ensure your InputFormat does not require one.
>     14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output
>     format specified. Ensure your OutputFormat does not require one.
>     14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom
>     argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapreduce.job.counters.limit is deprecated. Instead, use
>     mapreduce.job.counters.max
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapred.job.map.memory.mb is deprecated. Instead, use
>     mapreduce.map.memory.mb
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapred.job.reduce.memory.mb is deprecated. Instead, use
>     mapreduce.reduce.memory.mb
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapred.map.tasks.speculative.execution is deprecated. Instead, use
>     mapreduce.map.speculative
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapreduce.user.classpath.first is deprecated. Instead, use
>     mapreduce.job.user.classpath.first
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapred.map.max.attempts is deprecated. Instead, use
>     mapreduce.map.maxattempts
>     14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is
>     disabled (default), do not allow any task retries (setting
>     mapred.map.max.attempts = 0, old value = 4)
>     14/08/11 18:48:40 INFO Configuration.deprecation:
>     mapred.job.tracker is deprecated. Instead, use
>     mapreduce.jobtracker.address
>     14/08/11 18:48:41 INFO Configuration.deprecation: session.id
>     <http://session.id> is deprecated. Instead, use dfs.metrics.session-id
>     14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics
>     with processName=JobTracker, sessionId=
>     Exception in thread "main" java.lang.IncompatibleClassChangeError:
>     Found interface org.apache.hadoop.mapreduce.JobContext, but class
>     was expected
>             at
>     org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44)
>             at
>     org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
>             at
>     org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:422)
>             at
>     org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
>             at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>             at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>             at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>             at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>             at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>             at
>     sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>             at
>     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>             at java.lang.reflect.Method.invoke(Method.java:483)
>             at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>     Regards,
>     Vikalp Handa
>
>


Mime
View raw message