incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Can't get Getting Started example to work
Date Wed, 07 Sep 2011 21:31:56 GMT
Hi Kyle,

Thanks for your question and welcome to Giraph!  It looks like you 
couldn't get enough resources for the test to run on your hadoop 
instance.  In this example, you are asking for 30 workers.  You will 
need to be able to get 30 + 1 (master) = 31 map tasks to start the 
test.  If Giraph can't get all 31 map tasks within a period of time, it 
will fail.  Are you submitting this to an actual Hadoop cluster with at 
least 31 available map tasks?

Avery

On 9/7/11 2:13 PM, Kyle Teague wrote:
> I am trying to run the following command in pseudo-distributed mode
> from the Getting Started example page: hadoop jar
> giraph-0.70-jar-with-dependencies.jar
> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
> -w 30
>
> Here is the task log output:
>
> 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2011-09-07 15:41:34,529 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
> already exists!
> 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat:
> getOutputCommitter: Returning ImmutableOutputCommiter (does nothing).
> 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper:
> setup: jar file @
> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
> using /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar
> 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager:
> createCandidateStamp: Made the directory
> _bsp/_defaultZkManagerDir/job_201109071501_0003
> 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager:
> createCandidateStamp: Creating my filestamp
> _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home
> 0
> 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager:
> getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1
> candidates when 1 required (polling period is 3000) on attempt 0
> 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager:
> createZooKeeperServerList: Creating the final ZooKeeper file
> '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home
> 0 '
> 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager:
> getZooKeeperServerList: For task 0, got file
> 'zkServerList_new-host-3.home 0 ' (polling period is 3000)
> 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager:
> getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename
> 'zkServerList_new-host-3.home 0'
> 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Trying to delete old directory
> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
> generateZooKeeperConfigFile: Creating file
> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg
> in /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
> with base port 22181
> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
> generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true
> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
> generateZooKeeperConfigFile: Delete of zoo.cfg = false
> 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Attempting to start ZooKeeper server with
> command [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java,
> -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC,
> -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp,
> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
> org.apache.zookeeper.server.quorum.QuorumPeerMain,
> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg]
> in directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
> 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect
> to new-host-3.home:22181 with poll msecs = 3000
> 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Got ConnectException
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> 	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> 	at java.net.Socket.connect(Socket.java:529)
> 	at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:611)
> 	at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:419)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:253)
> 2011-09-07 15:41:37,776 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect
> to new-host-3.home:22181 with poll msecs = 3000
> 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Connected to
> new-host-3.home/192.168.1.6:22181!
> 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Creating my filestamp
> _bsp/_defaultZkManagerDir/job_201109071501_0003/_zkServer/new-host-3.home
> 0
> 2011-09-07 15:41:37,782 INFO org.apache.giraph.graph.GraphMapper:
> setup: Starting up BspServiceMaster (master thread)...
> 2011-09-07 15:41:37,791 INFO org.apache.giraph.graph.BspService:
> BspService: Connecting to ZooKeeper with job job_201109071501_0003, 0
> on new-host-3.home:22181
> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
> GMT
> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=new-host-3.home
> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_26
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Apple Inc.
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/classes:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work:/Users/kyle/hadoop/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Home//lib/tools.jar:/Users/kyle/hadoop/bin/..:/Users/kyle/hadoop/bin/../hadoop-core-0.20.203.0.jar:/Users/kyle/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/aspectjtools-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:/Users/kyle/hadoop/bin/../lib/commons-cli-1.2.jar:/Users/kyle/hadoop/bin/../lib/commons-codec-1.4.jar:/Users/kyle/hadoop/bin/../lib/commons-collections-3.2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-configuration-1.6.jar:/Users/kyle/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-digester-1.8.jar:/Users/kyle/hadoop/bin/../lib/commons-el-1.0.jar:/Users/kyle/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-lang-2.4.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-1.1.1.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/Users/kyle/hadoop/bin/../lib/commons-math-2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-net-1.4.1.jar:/Users/kyle/hadoop/bin/../lib/core-3.1.1.jar:/Users/kyle/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/Users/kyle/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jets3t-0.6.1.jar:/Users/kyle/hadoop/bin/../lib/jetty-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jetty-util-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jsch-0.1.42.jar:/Users/kyle/hadoop/bin/../lib/junit-4.5.jar:/Users/kyle/hadoop/bin/../lib/kfs-0.2.2.jar:/Users/kyle/hadoop/bin/../lib/log4j-1.2.15.jar:/Users/kyle/hadoop/bin/../lib/mockito-all-1.8.5.jar:/Users/kyle/hadoop/bin/../lib/oro-2.0.8.jar:/Users/kyle/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/Users/kyle/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/xmlenc-0.52.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.library.path=/Users/kyle/hadoop/bin/../lib/native/Mac_OS_X-x86_64-64:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work/tmp
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Mac OS X
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=x86_64
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=10.6.8
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=kyle
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/homes/
> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/private/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
> 2011-09-07 15:41:37,799 INFO org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=new-host-3.home:22181
> sessionTimeout=60000
> watcher=org.apache.giraph.graph.BspServiceMaster@769aba32
> 2011-09-07 15:41:37,810 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server new-host-3.home/192.168.1.6:22181
> 2011-09-07 15:41:37,811 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to new-host-3.home/192.168.1.6:22181,
> initiating session
> 2011-09-07 15:41:37,855 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server new-host-3.home/192.168.1.6:22181,
> sessionid = 0x1324568e60f0000, negotiated timeout = 60000
> 2011-09-07 15:41:37,856 INFO org.apache.giraph.graph.BspService:
> process: Asynchronous connection complete.
> 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper: map:
> No need to do anything when not a worker
> 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper:
> cleanup: Starting for MASTER_ZOOKEEPER_ONLY
> 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
> becomeMaster: First child is
> '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
> and my bid is '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
> 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
> becomeMaster: I am now the master!
> 2011-09-07 15:41:37,918 INFO org.apache.giraph.graph.BspService:
> process: applicationAttemptChanged signaled
> 2011-09-07 15:41:37,926 WARN org.apache.giraph.graph.BspService:
> process: Unknown and unprocessed event
> (path=/_hadoopBsp/job_201109071501_0003/_applicationAttemptsDir/0/_superstepDir,
> type=NodeChildrenChanged, state=SyncConnected)
> 2011-09-07 15:42:10,510 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 0 of 10 attempts.
> 2011-09-07 15:42:40,514 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 1 of 10 attempts.
> 2011-09-07 15:43:10,519 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 2 of 10 attempts.
> 2011-09-07 15:43:40,523 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 3 of 10 attempts.
> 2011-09-07 15:44:10,527 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 4 of 10 attempts.
> 2011-09-07 15:44:40,533 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 5 of 10 attempts.
> 2011-09-07 15:45:10,537 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 6 of 10 attempts.
> 2011-09-07 15:45:40,541 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 7 of 10 attempts.
> 2011-09-07 15:46:10,545 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 8 of 10 attempts.
> 2011-09-07 15:46:40,550 INFO org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Only found 1 responses of 30 needed to start superstep
> -1.  Sleeping for 30000 msecs and used 9 of 10 attempts.
> 2011-09-07 15:46:40,550 WARN org.apache.giraph.graph.BspServiceMaster:
> checkWorkers: Did not receive enough processes in time (only 1 of 30
> required)
> 2011-09-07 15:46:40,552 INFO org.apache.giraph.graph.BspServiceMaster:
> setJobState: {"_stateKey":"FAILED","_applicationAttemptKey":-1,"_superstepKey":-1}
> on superstep -1
> 2011-09-07 15:46:41,344 FATAL
> org.apache.giraph.graph.BspServiceMaster: failJob: Killing job
> job_201109071501_0003
> 2011-09-07 15:46:41,378 ERROR org.apache.giraph.graph.MasterThread:
> masterThread: Master algorithm failed:
> java.lang.NullPointerException
> 	at org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
> 	at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
> 2011-09-07 15:46:41,379 FATAL org.apache.giraph.graph.GraphMapper:
> uncaughtException: OverrideExceptionHandler on thread
> org.apache.giraph.graph.MasterThread, msg =
> java.lang.NullPointerException, exiting...
> java.lang.RuntimeException: java.lang.NullPointerException
> 	at org.apache.giraph.graph.MasterThread.run(MasterThread.java:177)
> Caused by: java.lang.NullPointerException
> 	at org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
> 	at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
> 2011-09-07 15:46:41,379 WARN org.apache.giraph.zk.ZooKeeperManager:
> onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper
> process.



Mime
View raw message