giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikalp Handa <handa.vik...@gmail.com>
Subject Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Date Tue, 12 Aug 2014 13:52:31 GMT
Hi Alexander,

I looked into my gam-stderr.log file and found *Error: Could not find or
load main class org.apache.giraph.yarn.GiraphApplicationMaster*
As you have already mentioned in your reply about that for loop in
*hadoop-env.sh. *
So can you please tell me what is *mylib *and *share/myLib/giraph** ? *Because
I am having only doc and hadoop directories inside  $HADOOP_HOME/share/

Also how to get *giraph-1.1.0-hadoop-2.4.0.jar *and*
giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the dependencies) as I am
having only giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar
inside giraph-examples/target/munged/.




Regards,
Vikalp Handa


On Tue, Aug 12, 2014 at 4:25 PM, Alexander Sirotin <sirotin8@web.de> wrote:

>  Hello Vikalp,
>
> you are showing the output of the client. The errors I ve showed were
> coming from the ResourceManager, the GiraphApplicationMaster
> (gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these
> logfiles on your systems.
>
> In case you get an error-message "class GiraphApplicationMaster not
> found": I solved it by adding a for-loop in hadoop-env.sh:
>
> for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do
>   if [ "$HADOOP_CLASSPATH" ]; then
>     export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
>   else
>     export HADOOP_CLASSPATH=$f
>   fi
> done
>
> In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and
> giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies).
> Additionally I copied every jar from /usr/local/giraph/lib also to this
> folder.
>
> Then you can restart the cluster and every instance should know giraph.
>
> After this it should function, if you have enough memory. I get then a
> message, that the GiraphApplicationMaster dont have enough memory and then
> get killed. But before this, it has started the TaskRunner, which keeps the
> whole job endlessly alive.
>
> Sincerely,
>
>
>
>
> On 12.08.2014 11:55, Vikalp Handa wrote:
>
> @Alexander Sirotin : Thanks for your reply. I am really sorry I haven't
> faced this problem after I executed it. Rather I am now having a different
> issue with Containers :
>
>  *Result : *
>  14/08/12 15:17:51 INFO yarn.GiraphYarnClient:
> ApplicationSumbissionContext for GiraphApplicationMaster launch container
> is populated.
> 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application
> application_1407836750214_0001
> 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after
> submission :application_1407836750214_0001
> 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99
> secs
> 14/08/12 15:17:52 INFO yarn.GiraphYarnClient:
> appattempt_1407836750214_0001_000001, State: ACCEPTED, Containers used: 1
> 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01
> secs
> 14/08/12 15:17:56 INFO yarn.GiraphYarnClient:
> appattempt_1407836750214_0001_000002, State: ACCEPTED, Containers used: 0
> 14/08/12 15:18:00* ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1407836750214_0001 failed
> 2 times due to AM Container for appattempt_1407836750214_0001_000002 exited
> with  exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: *
> *org.apache.hadoop.util.Shell$ExitCodeException:*
>  at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>  at org.apache.hadoop.util.Shell.run(Shell.java:418)
>  at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>  at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>  at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>  at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>
>
>  Container exited with a non-zero exit code 1
> .Failing this attempt.. Failing the application.
> 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
> cache directory for Giraph job.
> 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 7 seconds.
>
>  I have also checked my *yarn-site.xml* file and updated with the
> following property - value pairs inside configuration:
>
>    <property>
>         <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>      </property>
>      <property>
>         <name>yarn.application.classpath</name>
>         <value>
>              %HADOOP_HOME%\etc\hadoop,
>              %HADOOP_HOME%\share\hadoop\common\*,
>              %HADOOP_HOME%\share\hadoop\common\lib\*,
>              %HADOOP_HOME%\share\hadoop\hdfs\*,
>              %HADOOP_HOME%\share\hadoop\hdfs\lib\*,
>              %HADOOP_HOME%\share\hadoop\mapreduce\*,
>              %HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
>              %HADOOP_HOME%\share\hadoop\yarn\*,
>              %HADOOP_HOME%\share\hadoop\yarn\lib\*
>         </value>
>   </property>
>
>
>
>  Regards,
> Vikalp Handa
>
>
>

Mime
View raw message