giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Can't get Getting Started example to work
Date Thu, 08 Sep 2011 04:21:57 GMT
Glad to hear that got resolved Kyle.

Avery

On 9/7/11 7:54 PM, Kyle Teague wrote:
> Thanks! I don't have access to a full fledged Hadoop cluster right now
> -- just trying to test out the software on a single machine.  I
> changed the number of workers to 3 as I have one Task Tracker with a
> maximum of 4 map tasks and reduced the number of vertices to 500,000
> and that fixed it.
>
> I changed the number of workers to 2, which
> On Wed, Sep 7, 2011 at 5:31 PM, Avery Ching<aching@apache.org>  wrote:
>> Hi Kyle,
>>
>> Thanks for your question and welcome to Giraph!  It looks like you couldn't
>> get enough resources for the test to run on your hadoop instance.  In this
>> example, you are asking for 30 workers.  You will need to be able to get 30
>> + 1 (master) = 31 map tasks to start the test.  If Giraph can't get all 31
>> map tasks within a period of time, it will fail.  Are you submitting this to
>> an actual Hadoop cluster with at least 31 available map tasks?
>>
>> Avery
>>
>> On 9/7/11 2:13 PM, Kyle Teague wrote:
>>> I am trying to run the following command in pseudo-distributed mode
>>> from the Getting Started example page: hadoop jar
>>> giraph-0.70-jar-with-dependencies.jar
>>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>>> -w 30
>>>
>>> Here is the task log output:
>>>
>>> 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2011-09-07 15:41:34,529 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
>>> already exists!
>>> 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat:
>>> getOutputCommitter: Returning ImmutableOutputCommiter (does nothing).
>>> 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper:
>>> setup: jar file @
>>>
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
>>> using
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar
>>> 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> createCandidateStamp: Made the directory
>>> _bsp/_defaultZkManagerDir/job_201109071501_0003
>>> 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> createCandidateStamp: Creating my filestamp
>>> _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home
>>> 0
>>> 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1
>>> candidates when 1 required (polling period is 3000) on attempt 0
>>> 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> createZooKeeperServerList: Creating the final ZooKeeper file
>>>
>>> '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home
>>> 0 '
>>> 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> getZooKeeperServerList: For task 0, got file
>>> 'zkServerList_new-host-3.home 0 ' (polling period is 3000)
>>> 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename
>>> 'zkServerList_new-host-3.home 0'
>>> 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Trying to delete old directory
>>>
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
>>> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> generateZooKeeperConfigFile: Creating file
>>>
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg
>>> in
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
>>> with base port 22181
>>> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true
>>> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> generateZooKeeperConfigFile: Delete of zoo.cfg = false
>>> 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Attempting to start ZooKeeper server with
>>> command
>>> [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java,
>>> -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC,
>>> -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp,
>>>
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar,
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain,
>>>
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg]
>>> in directory
>>> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper
>>> 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect
>>> to new-host-3.home:22181 with poll msecs = 3000
>>> 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Got ConnectException
>>> java.net.ConnectException: Connection refused
>>>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>         at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>>         at
>>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>>>         at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>>>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>>>         at java.net.Socket.connect(Socket.java:529)
>>>         at
>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:611)
>>>         at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:419)
>>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>>         at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>> 2011-09-07 15:41:37,776 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect
>>> to new-host-3.home:22181 with poll msecs = 3000
>>> 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Connected to
>>> new-host-3.home/192.168.1.6:22181!
>>> 2011-09-07 15:41:37,777 INFO org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Creating my filestamp
>>> _bsp/_defaultZkManagerDir/job_201109071501_0003/_zkServer/new-host-3.home
>>> 0
>>> 2011-09-07 15:41:37,782 INFO org.apache.giraph.graph.GraphMapper:
>>> setup: Starting up BspServiceMaster (master thread)...
>>> 2011-09-07 15:41:37,791 INFO org.apache.giraph.graph.BspService:
>>> BspService: Connecting to ZooKeeper with job job_201109071501_0003, 0
>>> on new-host-3.home:22181
>>> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>>> GMT
>>> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:host.name=new-host-3.home
>>> 2011-09-07 15:41:37,797 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:java.version=1.6.0_26
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:java.vendor=Apple Inc.
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>>
>>> environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>>
>>> environment:java.class.path=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/classes:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work:/Users/kyle/hadoop/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Home//lib/tools.jar:/Users/kyle/hadoop/bin/..:/Users/kyle/hadoop/bin/../hadoop-core-0.20.203.0.jar:/Users/kyle/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/aspectjtools-1.6.5.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/Users/kyle/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:/Users/kyle/hadoop/bin/../lib/commons-cli-1.2.jar:/Users/kyle/hadoop/bin/../lib/commons-codec-1.4.jar:/Users/kyle/hadoop/bin/../lib/commons-collections-3.2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-configuration-1.6.jar:/Users/kyle/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-digester-1.8.jar:/Users/kyle/hadoop/bin/../lib/commons-el-1.0.jar:/Users/kyle/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/Users/kyle/hadoop/bin/../lib/commons-lang-2.4.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-1.1.1.jar:/Users/kyle/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/Users/kyle/hadoop/bin/../lib/commons-math-2.1.jar:/Users/kyle/hadoop/bin/../lib/commons-net-1.4.1.jar:/Users/kyle/hadoop/bin/../lib/core-3.1.1.jar:/Users/kyle/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/Users/kyle/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/Users/kyle/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/Users/kyle/hadoop/bin/../lib/jets3t-0.6.1.jar:/Users/kyle/hadoop/bin/../lib/jetty-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jetty-util-6.1.26.jar:/Users/kyle/hadoop/bin/../lib/jsch-0.1.42.jar:/Users/kyle/hadoop/bin/../lib/junit-4.5.jar:/Users/kyle/hadoop/bin/../lib/kfs-0.2.2.jar:/Users/kyle/hadoop/bin/../lib/log4j-1.2.15.jar:/Users/kyle/hadoop/bin/../lib/mockito-all-1.8.5.jar:/Users/kyle/hadoop/bin/../lib/oro-2.0.8.jar:/Users/kyle/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/Users/kyle/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/Users/kyle/hadoop/bin/../lib/xmlenc-0.52.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/Users/kyle/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>>
>>> environment:java.library.path=/Users/kyle/hadoop/bin/../lib/native/Mac_OS_X-x86_64-64:/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>>
>>> environment:java.io.tmpdir=/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work/tmp
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:java.compiler=<NA>
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:os.name=Mac OS X
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:os.arch=x86_64
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:os.version=10.6.8
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:user.name=kyle
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>> environment:user.home=/homes/
>>> 2011-09-07 15:41:37,798 INFO org.apache.zookeeper.ZooKeeper: Client
>>>
>>> environment:user.dir=/private/tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/attempt_201109071501_0003_m_000000_0/work
>>> 2011-09-07 15:41:37,799 INFO org.apache.zookeeper.ZooKeeper:
>>> Initiating client connection, connectString=new-host-3.home:22181
>>> sessionTimeout=60000
>>> watcher=org.apache.giraph.graph.BspServiceMaster@769aba32
>>> 2011-09-07 15:41:37,810 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket connection to server new-host-3.home/192.168.1.6:22181
>>> 2011-09-07 15:41:37,811 INFO org.apache.zookeeper.ClientCnxn: Socket
>>> connection established to new-host-3.home/192.168.1.6:22181,
>>> initiating session
>>> 2011-09-07 15:41:37,855 INFO org.apache.zookeeper.ClientCnxn: Session
>>> establishment complete on server new-host-3.home/192.168.1.6:22181,
>>> sessionid = 0x1324568e60f0000, negotiated timeout = 60000
>>> 2011-09-07 15:41:37,856 INFO org.apache.giraph.graph.BspService:
>>> process: Asynchronous connection complete.
>>> 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper: map:
>>> No need to do anything when not a worker
>>> 2011-09-07 15:41:37,857 INFO org.apache.giraph.graph.GraphMapper:
>>> cleanup: Starting for MASTER_ZOOKEEPER_ONLY
>>> 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
>>> becomeMaster: First child is
>>>
>>> '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
>>> and my bid is
>>> '/_hadoopBsp/job_201109071501_0003/_masterElectionDir/new-host-3.home_00000000000'
>>> 2011-09-07 15:41:37,907 INFO org.apache.giraph.graph.BspServiceMaster:
>>> becomeMaster: I am now the master!
>>> 2011-09-07 15:41:37,918 INFO org.apache.giraph.graph.BspService:
>>> process: applicationAttemptChanged signaled
>>> 2011-09-07 15:41:37,926 WARN org.apache.giraph.graph.BspService:
>>> process: Unknown and unprocessed event
>>>
>>> (path=/_hadoopBsp/job_201109071501_0003/_applicationAttemptsDir/0/_superstepDir,
>>> type=NodeChildrenChanged, state=SyncConnected)
>>> 2011-09-07 15:42:10,510 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 0 of 10 attempts.
>>> 2011-09-07 15:42:40,514 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 1 of 10 attempts.
>>> 2011-09-07 15:43:10,519 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 2 of 10 attempts.
>>> 2011-09-07 15:43:40,523 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 3 of 10 attempts.
>>> 2011-09-07 15:44:10,527 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 4 of 10 attempts.
>>> 2011-09-07 15:44:40,533 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 5 of 10 attempts.
>>> 2011-09-07 15:45:10,537 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 6 of 10 attempts.
>>> 2011-09-07 15:45:40,541 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 7 of 10 attempts.
>>> 2011-09-07 15:46:10,545 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 8 of 10 attempts.
>>> 2011-09-07 15:46:40,550 INFO org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Only found 1 responses of 30 needed to start superstep
>>> -1.  Sleeping for 30000 msecs and used 9 of 10 attempts.
>>> 2011-09-07 15:46:40,550 WARN org.apache.giraph.graph.BspServiceMaster:
>>> checkWorkers: Did not receive enough processes in time (only 1 of 30
>>> required)
>>> 2011-09-07 15:46:40,552 INFO org.apache.giraph.graph.BspServiceMaster:
>>> setJobState:
>>> {"_stateKey":"FAILED","_applicationAttemptKey":-1,"_superstepKey":-1}
>>> on superstep -1
>>> 2011-09-07 15:46:41,344 FATAL
>>> org.apache.giraph.graph.BspServiceMaster: failJob: Killing job
>>> job_201109071501_0003
>>> 2011-09-07 15:46:41,378 ERROR org.apache.giraph.graph.MasterThread:
>>> masterThread: Master algorithm failed:
>>> java.lang.NullPointerException
>>>         at
>>> org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
>>>         at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
>>> 2011-09-07 15:46:41,379 FATAL org.apache.giraph.graph.GraphMapper:
>>> uncaughtException: OverrideExceptionHandler on thread
>>> org.apache.giraph.graph.MasterThread, msg =
>>> java.lang.NullPointerException, exiting...
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>         at org.apache.giraph.graph.MasterThread.run(MasterThread.java:177)
>>> Caused by: java.lang.NullPointerException
>>>         at
>>> org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:486)
>>>         at org.apache.giraph.graph.MasterThread.run(MasterThread.java:94)
>>> 2011-09-07 15:46:41,379 WARN org.apache.giraph.zk.ZooKeeperManager:
>>> onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper
>>> process.
>>
>>



Mime
View raw message