Return-Path: X-Original-To: apmail-incubator-giraph-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-giraph-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DFEE78ED9 for ; Wed, 7 Sep 2011 21:31:57 +0000 (UTC) Received: (qmail 32748 invoked by uid 500); 7 Sep 2011 21:31:57 -0000 Delivered-To: apmail-incubator-giraph-user-archive@incubator.apache.org Received: (qmail 32723 invoked by uid 500); 7 Sep 2011 21:31:57 -0000 Mailing-List: contact giraph-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: giraph-user@incubator.apache.org Delivered-To: mailing list giraph-user@incubator.apache.org Received: (qmail 32713 invoked by uid 99); 7 Sep 2011 21:31:57 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Sep 2011 21:31:57 +0000 Received: from localhost (HELO [0.0.0.0]) (127.0.0.1) (smtp-auth username aching, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Sep 2011 21:31:57 +0000 Message-ID: <4E67E2CC.2020000@apache.org> Date: Wed, 07 Sep 2011 14:31:56 -0700 From: Avery Ching User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2 MIME-Version: 1.0 To: giraph-user@incubator.apache.org CC: Kyle Teague Subject: Re: Can't get Getting Started example to work References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Hi Kyle, Thanks for your question and welcome to Giraph! It looks like you=20 couldn't get enough resources for the test to run on your hadoop=20 instance. In this example, you are asking for 30 workers. You will=20 need to be able to get 30 + 1 (master) =3D 31 map tasks to start the=20 test. If Giraph can't get all 31 map tasks within a period of time, it=20 will fail. Are you submitting this to an actual Hadoop cluster with at=20 least 31 available map tasks? Avery On 9/7/11 2:13 PM, Kyle Teague wrote: > I am trying to run the following command in pseudo-distributed mode > from the Getting Started example page: hadoop jar > giraph-0.70-jar-with-dependencies.jar > org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 > -w 30 > > Here is the task log output: > > 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > 2011-09-07 15:41:34,529 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi > already exists! > 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat: > getOutputCommitter: Returning ImmutableOutputCommiter (does nothing). > 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper: > setup: jar file @ > /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907150= 1_0003/jars/job.jar, > using /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110= 9071501_0003/jars/job.jar > 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager: > createCandidateStamp: Made the directory > _bsp/_defaultZkManagerDir/job_201109071501_0003 > 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager: > createCandidateStamp: Creating my filestamp > _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home > 0 > 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager: > getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1 > candidates when 1 required (polling period is 3000) on attempt 0 > 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager: > createZooKeeperServerList: Creating the final ZooKeeper file > '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-= 3.home > 0 ' > 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager: > getZooKeeperServerList: For task 0, got file > 'zkServerList_new-host-3.home 0 ' (polling period is 3000) > 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager: > getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename > 'zkServerList_new-host-3.home 0' > 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Trying to delete old directory > /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907150= 1_0003/work/_bspZooKeeper > 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: > generateZooKeeperConfigFile: Creating file > /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907150= 1_0003/work/_bspZooKeeper/zoo.cfg > in /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907= 1501_0003/work/_bspZooKeeper > with base port 22181 > 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: > generateZooKeeperConfigFile: Make directory of _bspZooKeeper =3D true > 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: > generateZooKeeperConfigFile: Delete of zoo.cfg =3D false > 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Attempting to start ZooKeeper server with > command [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Ho= me/bin/java, > -Xmx256m, -XX:ParallelGCThreads=3D4, -XX:+UseConcMarkSweepGC, > -XX:CMSInitiatingOccupancyFraction=3D70, -XX:MaxGCPauseMillis=3D100, -c= p, > /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907150= 1_0003/jars/job.jar, > org.apache.zookeeper.server.quorum.QuorumPeerMain, > /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_20110907150= 1_0003/work/_bspZooKeeper/zoo.cfg] > in directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/jo= b_201109071501_0003/work/_bspZooKeeper > 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect > to new-host-3.home:22181 with poll msecs =3D 3000 > 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Got ConnectException > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)= > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432) > at java.net.Socket.connect(Socket.java:529) > at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKee= perManager.java:611) > at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:419) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor= mation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:253) > 2011-09-07 15:41:37,776 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect > to new-host-3.home:22181 with poll msecs =3D 3000 > 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Connected to > new-host-3.home/192.168.1.6:22181! > 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Creating my filestamp > _bsp/_defaultZkManagerDir/job_201109071501_0003/_zkServer/new-host-3.ho= me > 0 > 2011-09-07 15:41:37,782 INFO org.apache.giraph.graph.GraphMapper: > setup: Starting up BspServiceMaster (master thread)... > 2011-09-07 15:41:37,791 INFO org.apache.giraph.graph.BspService: > BspService: Connecting to ZooKeeper with job job_201109071501_0003, 0 > on new-host-3.home:22181 > 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client > environment:zookeeper.version=3D3.3.1-942149, built on 05/07/2010 17:14= > GMT > 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client > environment:host.name=3Dnew-host-3.home > 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.version=3D1.6.0_26 > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.vendor=3DApple Inc. > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.home=3D/System/Library/Java/JavaVirtualMachines/1.6.0.= jdk/Contents/Home > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.class.path=3D/tmp/hadoop-kyle/mapred/local/taskTracker= /kyle/jobcache/job_201109071501_0003/jars/classes:/tmp/hadoop-kyle/mapred= /local/taskTracker/kyle/jobcache/job_201109071501_0003/jars:/tmp/hadoop-k= yle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_= 201109071501_0003_m_000000_0/work:/Users/kyle/hadoop/bin/../conf:/System/= Library/Frameworks/JavaVM.framework/Home//lib/tools.jar:/Users/kyle/hadoo= p/bin/..:/Users/kyle/hadoop/bin/../hadoop-core-0.20.203.0.jar:/Users/kyle= /hadoop/bin/../lib/aspectjrt-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/aspe= ctjtools-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-1.7.0.= jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:/Users= /kyle/hadoop/bin/../lib/commons-cli-1.2.jar:/Users/kyle/hadoop/bin/../lib= /commons-codec-1.4.jar:/Users/kyle/hadoop/bin/../lib/commons-collections-= 3.2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-configuration-1.6.jar:/Us= ers/kyle/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/Users/kyle/hadoop/bi= n/../lib/commons-digester-1.8.jar:/Users/kyle/hadoop/bin/../lib/commons-e= l-1.0.jar:/Users/kyle/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/Use= rs/kyle/hadoop/bin/../lib/commons-lang-2.4.jar:/Users/kyle/hadoop/bin/../= lib/commons-logging-1.1.1.jar:/Users/kyle/hadoop/bin/../lib/commons-loggi= ng-api-1.0.4.jar:/Users/kyle/hadoop/bin/../lib/commons-math-2.1.jar:/User= s/kyle/hadoop/bin/../lib/commons-net-1.4.1.jar:/Users/kyle/hadoop/bin/../= lib/core-3.1.1.jar:/Users/kyle/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/Use= rs/kyle/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:/Users/kyle/hadoop/b= in/../lib/jackson-mapper-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jasp= er-compiler-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jasper-runtime-5.5.1= 2.jar:/Users/kyle/hadoop/bin/../lib/jets3t-0.6.1.jar:/Users/kyle/hadoop/b= in/../lib/jetty-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jetty-util-6.1.2= 6.jar:/Users/kyle/hadoop/bin/../lib/jsch-0.1.42.jar:/Users/kyle/hadoop/bi= n/../lib/junit-4.5.jar:/Users/kyle/hadoop/bin/../lib/kfs-0.2.2.jar:/Users= /kyle/hadoop/bin/../lib/log4j-1.2.15.jar:/Users/kyle/hadoop/bin/../lib/mo= ckito-all-1.8.5.jar:/Users/kyle/hadoop/bin/../lib/oro-2.0.8.jar:/Users/ky= le/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/Users/kyle/hadoop/bin/= =2E./lib/slf4j-api-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/slf4j-log4j12-= 1.4.3.jar:/Users/kyle/hadoop/bin/../lib/xmlenc-0.52.jar:/Users/kyle/hadoo= p/bin/../lib/jsp-2.1/jsp-2.1.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/js= p-api-2.1.jar > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.library.path=3D/Users/kyle/hadoop/bin/../lib/native/Ma= c_OS_X-x86_64-64:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/= job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=3D/tmp/hadoop-kyle/mapred/local/taskTracker/= kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/= work/tmp > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:java.compiler=3D > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:os.name=3DMac OS X > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:os.arch=3Dx86_64 > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:os.version=3D10.6.8 > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:user.name=3Dkyle > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:user.home=3D/homes/ > 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client > environment:user.dir=3D/private/tmp/hadoop-kyle/mapred/local/taskTracke= r/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_= 0/work > 2011-09-07 15:41:37,799 INFO org.apache.zookeeper.ZooKeeper: > Initiating client connection, connectString=3Dnew-host-3.home:22181 > sessionTimeout=3D60000 > watcher=3Dorg.apache.giraph.graph.BspServiceMaster@769aba32 > 2011-09-07 15:41:37,810 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server new-host-3.home/192.168.1.6:22181 > 2011-09-07 15:41:37,811 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to new-host-3.home/192.168.1.6:22181, > initiating session > 2011-09-07 15:41:37,855 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server new-host-3.home/192.168.1.6:22181, > sessionid =3D 0x1324568e60f0000, negotiated timeout =3D 60000 > 2011-09-07 15:41:37,856 INFO org.apache.giraph.graph.BspService: > process: Asynchronous connection complete. > 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper: map: > No need to do anything when not a worker > 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper: > cleanup: Starting for MASTER_ZOOKEEPER_ONLY > 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster: > becomeMaster: First child is > '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_0= 0000000000' > and my bid is '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new= -host-3.home_00000000000' > 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster: > becomeMaster: I am now the master! > 2011-09-07 15:41:37,918 INFO org.apache.giraph.graph.BspService: > process: applicationAttemptChanged signaled > 2011-09-07 15:41:37,926 WARN org.apache.giraph.graph.BspService: > process: Unknown and unprocessed event > (path=3D/_hadoopBsp/job_201109071501_0003/_applicationAttemptsDir/0/_su= perstepDir, > type=3DNodeChildrenChanged, state=3DSyncConnected) > 2011-09-07 15:42:10,510 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 0 of 10 attempts. > 2011-09-07 15:42:40,514 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 1 of 10 attempts. > 2011-09-07 15:43:10,519 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 2 of 10 attempts. > 2011-09-07 15:43:40,523 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 3 of 10 attempts. > 2011-09-07 15:44:10,527 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 4 of 10 attempts. > 2011-09-07 15:44:40,533 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 5 of 10 attempts. > 2011-09-07 15:45:10,537 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 6 of 10 attempts. > 2011-09-07 15:45:40,541 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 7 of 10 attempts. > 2011-09-07 15:46:10,545 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 8 of 10 attempts. > 2011-09-07 15:46:40,550 INFO org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Only found 1 responses of 30 needed to start superstep > -1. Sleeping for 30000 msecs and used 9 of 10 attempts. > 2011-09-07 15:46:40,550 WARN org.apache.giraph.graph.BspServiceMaster: > checkWorkers: Did not receive enough processes in time (only 1 of 30 > required) > 2011-09-07 15:46:40,552 INFO org.apache.giraph.graph.BspServiceMaster: > setJobState: {"_stateKey":"FAILED","_applicationAttemptKey":-1,"_supers= tepKey":-1} > on superstep -1 > 2011-09-07 15:46:41,344 FATAL > org.apache.giraph.graph.BspServiceMaster: failJob: Killing job > job_201109071501_0003 > 2011-09-07 15:46:41,378 ERROR org.apache.giraph.graph.MasterThread: > masterThread: Master algorithm failed: > java.lang.NullPointerException > at org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServi= ceMaster.java:486) > at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94) > 2011-09-07 15:46:41,379 FATAL org.apache.giraph.graph.GraphMapper: > uncaughtException: OverrideExceptionHandler on thread > org.apache.giraph.graph.MasterThread, msg =3D > java.lang.NullPointerException, exiting... > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.giraph.graph.MasterThread.run(MasterThread.java:177) > Caused by: java.lang.NullPointerException > at org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServi= ceMaster.java:486) > at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94) > 2011-09-07 15:46:41,379 WARN org.apache.giraph.zk.ZooKeeperManager: > onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper > process.