giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicola Barbieri <barbi...@yahoo-inc.com>
Subject Error on large graphs
Date Thu, 29 Oct 2015 16:33:45 GMT
Hi,
I am struggling with some exceptions when running my job on a large graph.The job runs correctly
on a toy example (both local & non-local mode). 
I am using the following settings:
-Dmapred.child.java.opts="-Xmx4g -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=100
-XX:-UseGCOverheadLimit" \
-Dmapreduce.map.memory.mb=8192 \-Dgiraph.zkList=[list_instances]\-Dgiraph.zkJavaOpts="-Xmx4g
-XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxGCPauseMillis=100"
\-Dgiraph.useOutOfCoreGraph=true \-Dgiraph.isStaticGraph=true \-Dgiraph.useOutOfCoreMessages=true
 \-Dgiraph.maxMessagesInMemory=10000 \-Dgiraph.partitionsDirectory="tmp/_bsp/_partitions"
\-Dgiraph.messagesDirectory="tmp/_bsp/_messages/"  \-w 300 

The job fails early if I set giraph.maxPartitionsInMemory. 
Here is the log of the error:
015-10-29 15:50:53,064 ERROR [org.apache.giraph.master.MasterThread] org.apache.giraph.master.MasterThread:
masterThread: Master algorithm failed with RuntimeExceptionjava.lang.RuntimeException: java.io.IOException:
Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "nodename.com/xx.xxx.xxx.xx";
destination host is: "nodename.com":8032;  at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:382)
at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:311)
at org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1358)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1592)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)Caused by: java.io.IOException:
Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "nodename.com/xx.xxx.xxx.xx";
destination host is: "nodename.com":8032;  at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:357)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694) at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596)
at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:374) ... 4 more2015-10-29
15:50:53,067 FATAL [org.apache.giraph.master.MasterThread] org.apache.giraph.graph.GraphTaskManager:
uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread,
msg = java.lang.RuntimeException: java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN,
KERBEROS]; Host Details : local host is: "nodename.com/10.216.170.23"; destination host is:
"nodename.com":8032; , exiting...java.lang.IllegalStateException: java.lang.RuntimeException:
java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "nodename.com/xx.xx.xxx.xx";
destination host is: "nodename.com":8032;  at org.apache.giraph.master.MasterThread.run(MasterThread.java:194)Caused
by: java.lang.RuntimeException: java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN,
KERBEROS]; Host Details : local host is: "nodename.com/xx.xxx.xxx.xx"; destination host is:
"nodename.com":8032;  at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:382)
at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:311)
at org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1358)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1592)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)Caused by: java.io.IOException:
Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "nodename.com/xx.xxx.xxx.xx";
destination host is: "nodename.com":8032;  at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:357)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694) at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596)
at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:374) ... 4 more

General question:Is it mandatory to specify some Zookeper instances (-Dgiraph.zkList) or in
case nothing is specified Giraph will start a new one? 

Mime
View raw message