giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: A simple use case: shortest paths on a FOAF (i.e. Friend of a Friend) graph
Date Fri, 13 Apr 2012 16:34:26 GMT
Hi Paulo,

Can you try something for me?  I was able to get the PageRankBenchmark 
to work running in local mode just fine on my side.

I think we should have some kind of a helper script (similar to 
bin/giraph) to running simple tests in LocalJobRunner.

I believe that for LocalJobRunner to run, we need to do 
-Dgiraph.SplitMasterWorker=false -Dlocal.test.mode=true.  In the case of 
PageRankBenchmark, I also have to set the workers to 1 (LocalJobRunner 
can only run one task at a time).

So I get the class path that bin/giraph was using to run (just added a 
echo $CLASSPATH at the end) and then inserted the 
giraph-0.2-SNAPSHOT-jar-with-dependencies.jar in front of it (this is 
necessary for the ZooKeeper jar inclusion).  Then I just ran a normal 
java command and the output below.

One thing to remember is that if you rerun it, you'll have to remove the 
_bsp directories that are created, otherwise it will think it has 
already been completed.

Hope that helps,

Avery

  java -cp 
target/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar:/Users/aching/git/git_svn_giraph_trunk/conf:/Users/aching/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:/Users/aching/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/aching/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/aching/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/aching/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/aching/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/aching/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/aching/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/aching/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/aching/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/aching/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/Users/aching/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/aching/.m2/repository/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.jar:/Users/aching/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/aching/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar:/Users/aching/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/aching/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/aching/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/aching/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/aching/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/aching/.m2/repository/net/iharder/base64/2.3.8/base64-2.3.8.jar:/Users/aching/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:/Users/aching/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar:/Users/aching/.m2/repository/org/apache/commons/commons-io/1.3.2/commons-io-1.3.2.jar:/Users/aching/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/aching/.m2/repository/org/apache/hadoop/hadoop-core/0.20.203.0/hadoop-core-0.20.203.0.jar:/Users/aching/.m2/repository/org/apache/mahout/mahout-collections/1.0/mahout-collections-1.0.jar:/Users/aching/.m2/repository/org/apache/zookeeper/zookeeper/3.3.3/zookeeper-3.3.3.jar:/Users/aching/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.0/jackson-core-asl-1.8.0.jar:/Users/aching/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.0/jackson-mapper-asl-1.8.0.jar:/Users/aching/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/aching/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/aching/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/aching/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar:/Users/aching/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/aching/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/aching/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/aching/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/aching/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar

org.apache.giraph.benchmark.PageRankBenchmark 
-Dgiraph.SplitMasterWorker=false -Dlocal.test.mode=true  -c 1 -e 2 -s 2 
-V 10 -w 1

2012-04-13 09:30:27.261 java[45785:1903] Unable to load realm mapping 
info from SCDynamicStore
12/04/13 09:30:27 INFO benchmark.PageRankBenchmark: Using class 
org.apache.giraph.benchmark.PageRankBenchmark
12/04/13 09:30:27 WARN bsp.BspOutputFormat: checkOutputSpecs: 
ImmutableOutputCommiter will not check anything
12/04/13 09:30:27 WARN bsp.BspOutputFormat: getOutputCommitter: 
Returning ImmutableOutputCommiter (does nothing).
12/04/13 09:30:27 INFO graph.GraphMapper: Distributed cache is empty. 
Assuming fatjar.
12/04/13 09:30:27 INFO graph.GraphMapper: setup: classpath @ 
/tmp/hadoop-aching/mapred/staging/aching-221077155/.staging/job_local_0001/job.jar
12/04/13 09:30:27 INFO zk.ZooKeeperManager: createCandidateStamp: Made 
the directory _bsp/_defaultZkManagerDir/job_local_0001
12/04/13 09:30:27 INFO zk.ZooKeeperManager: createCandidateStamp: 
Creating my filestamp 
_bsp/_defaultZkManagerDir/job_local_0001/_task/192.168.1.100 0
12/04/13 09:30:27 INFO zk.ZooKeeperManager: getZooKeeperServerList: Got 
[192.168.1.100] 1 hosts from 1 candidates when 1 required (polling 
period is 3000) on attempt 0
12/04/13 09:30:27 INFO zk.ZooKeeperManager: createZooKeeperServerList: 
Creating the final ZooKeeper file 
'_bsp/_defaultZkManagerDir/job_local_0001/zkServerList_192.168.1.100 0 '
12/04/13 09:30:27 INFO zk.ZooKeeperManager: getZooKeeperServerList: For 
task 0, got file 'zkServerList_192.168.1.100 0 ' (polling period is 3000)
12/04/13 09:30:27 INFO zk.ZooKeeperManager: getZooKeeperServerList: 
Found [192.168.1.100, 0] 2 hosts in filename 'zkServerList_192.168.1.100 0 '
12/04/13 09:30:27 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Trying to delete old directory 
/Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper
12/04/13 09:30:27 INFO zk.ZooKeeperManager: generateZooKeeperConfigFile: 
Creating file 
/Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper/zoo.cfg in 
/Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper with base port 22181
12/04/13 09:30:27 INFO zk.ZooKeeperManager: generateZooKeeperConfigFile: 
Make directory of _bspZooKeeper = true
12/04/13 09:30:27 INFO zk.ZooKeeperManager: generateZooKeeperConfigFile: 
Delete of zoo.cfg = false
12/04/13 09:30:27 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Attempting to start ZooKeeper server with command 
[/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java, 
-Xmx512m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC, 
-XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp, 
/tmp/hadoop-aching/mapred/staging/aching-221077155/.staging/job_local_0001/job.jar, 
org.apache.zookeeper.server.quorum.QuorumPeerMain, 
/Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper/zoo.cfg] in 
directory /Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper
12/04/13 09:30:27 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Connect attempt 0 of 10 max trying to connect to 192.168.1.100:22181 
with poll msecs = 3000
12/04/13 09:30:27 WARN zk.ZooKeeperManager: onlineZooKeeperServers: Got 
ConnectException
java.net.ConnectException: Connection refused
     at java.net.PlainSocketImpl.socketConnect(Native Method)
     at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
     at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
     at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
     at java.net.Socket.connect(Socket.java:529)
     at 
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:661)
     at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:425)
     at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:646)
     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
     at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
12/04/13 09:30:30 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Connect attempt 1 of 10 max trying to connect to 192.168.1.100:22181 
with poll msecs = 3000
12/04/13 09:30:30 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Connected to /192.168.1.100:22181!
12/04/13 09:30:30 INFO zk.ZooKeeperManager: onlineZooKeeperServers: 
Creating my filestamp 
_bsp/_defaultZkManagerDir/job_local_0001/_zkServer/192.168.1.100 0
12/04/13 09:30:30 INFO graph.GraphMapper: setup: Starting up 
BspServiceMaster (master thread)...
12/04/13 09:30:30 INFO graph.BspService: BspService: Connecting to 
ZooKeeper with job job_local_0001, 0 on 192.168.1.100:22181
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:host.name=192.168.1.100
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.version=1.6.0_31
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.vendor=Apple Inc.
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.class.path=target/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar:/Users/aching/git/git_svn_giraph_trunk/conf:/Users/aching/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:/Users/aching/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/aching/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/aching/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/aching/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/aching/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/aching/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/aching/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/aching/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/aching/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/aching/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/Users/aching/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/aching/.m2/repository/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.jar:/Users/aching/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/aching/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar:/Users/aching/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/aching/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/aching/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/aching/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/aching/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/aching/.m2/repository/net/iharder/base64/2.3.8/base64-2.3.8.jar:/Users/aching/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:/Users/aching/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar:/Users/aching/.m2/repository/org/apache/commons/commons-io/1.3.2/commons-io-1.3.2.jar:/Users/aching/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/aching/.m2/repository/org/apache/hadoop/hadoop-core/0.20.203.0/hadoop-core-0.20.203.0.jar:/Users/aching/.m2/repository/org/apache/mahout/mahout-collections/1.0/mahout-collections-1.0.jar:/Users/aching/.m2/repository/org/apache/zookeeper/zookeeper/3.3.3/zookeeper-3.3.3.jar:/Users/aching/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.0/jackson-core-asl-1.8.0.jar:/Users/aching/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.0/jackson-mapper-asl-1.8.0.jar:/Users/aching/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/aching/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/aching/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/aching/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/aching/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar:/Users/aching/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/aching/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/aching/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/aching/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/aching/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.io.tmpdir=/var/folders/ks/2sm0ygc947x00lmprdkbkhqhb4b6h3/T/
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:java.compiler=<NA>
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:os.name=Mac OS X
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:os.arch=x86_64
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:os.version=10.7.3
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:user.name=aching
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:user.home=/Users/aching
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Client 
environment:user.dir=/Users/aching/git/git_svn_giraph_trunk
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Initiating client 
connection, connectString=192.168.1.100:22181 sessionTimeout=60000 
watcher=org.apache.giraph.graph.BspServiceMaster@5e2b323e
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Opening socket connection 
to server /192.168.1.100:22181
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Socket connection 
established to 192.168.1.100/192.168.1.100:22181, initiating session
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Session establishment 
complete on server 192.168.1.100/192.168.1.100:22181, sessionid = 
0x136ac8aa05e0000, negotiated timeout = 300000
12/04/13 09:30:30 INFO graph.BspService: process: Asynchronous 
connection complete.
12/04/13 09:30:30 INFO graph.GraphMapper: setup: Starting up 
BspServiceWorker...
12/04/13 09:30:30 INFO graph.BspService: BspService: Connecting to 
ZooKeeper with job job_local_0001, 0 on 192.168.1.100:22181
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Initiating client 
connection, connectString=192.168.1.100:22181 sessionTimeout=60000 
watcher=org.apache.giraph.graph.BspServiceWorker@2c31f2a7
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Opening socket connection 
to server /192.168.1.100:22181
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Socket connection 
established to 192.168.1.100/192.168.1.100:22181, initiating session
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: Session establishment 
complete on server 192.168.1.100/192.168.1.100:22181, sessionid = 
0x136ac8aa05e0001, negotiated timeout = 300000
12/04/13 09:30:30 INFO graph.BspService: process: Asynchronous 
connection complete.
12/04/13 09:30:30 INFO ipc.Server: Starting SocketReader
12/04/13 09:30:30 INFO ipc.Server: IPC Server Responder: starting
12/04/13 09:30:30 INFO ipc.Server: IPC Server listener on 30000: starting
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: 
BasicRPCCommunications: Started RPC communication server: 
achingmbp15.local/192.168.1.100:30000 with 1 handlers and 1 flush 
threads on bind attempt 0
12/04/13 09:30:30 INFO ipc.Server: IPC Server handler 0 on 30000: starting
12/04/13 09:30:30 INFO graph.GraphMapper: setup: Registering health of 
this worker...
12/04/13 09:30:30 INFO graph.BspService: getJobState: Job state already 
exists (/_hadoopBsp/job_local_0001/_masterJobState)
12/04/13 09:30:30 INFO graph.BspServiceMaster: becomeMaster: First child 
is 
'/_hadoopBsp/job_local_0001/_masterElectionDir/achingmbp15.local_00000000000' 
and my bid is 
'/_hadoopBsp/job_local_0001/_masterElectionDir/achingmbp15.local_00000000000'
12/04/13 09:30:30 INFO graph.BspService: getApplicationAttempt: Node 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
12/04/13 09:30:30 INFO graph.BspServiceMaster: becomeMaster: I am now 
the master!
12/04/13 09:30:30 INFO graph.BspService: process: 
applicationAttemptChanged signaled
12/04/13 09:30:30 INFO graph.BspService: process: 
applicationAttemptChanged signaled
12/04/13 09:30:30 INFO graph.BspService: getApplicationAttempt: Node 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
12/04/13 09:30:30 WARN graph.BspService: process: Unknown and 
unprocessed event 
(path=/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir, type=NodeChildrenChanged,

state=SyncConnected)
12/04/13 09:30:30 WARN graph.BspService: process: Unknown and 
unprocessed event 
(path=/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir, type=NodeChildrenChanged,

state=SyncConnected)
12/04/13 09:30:30 INFO graph.BspServiceWorker: registerHealth: Created 
my health node for attempt=0, superstep=-1 with 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir/achingmbp15.local_0

and workerInfo= Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000)
12/04/13 09:30:30 INFO graph.BspServiceMaster: generateInputSplits: Got 
1 input splits for 1 workers
12/04/13 09:30:30 INFO partition.HashMasterPartitioner: 
createInitialPartitionOwners: Creating 1, default would have been 1 
partitions.
12/04/13 09:30:30 INFO graph.BspService: process: 
partitionAssignmentsReadyChanged (partitions are assigned)
12/04/13 09:30:30 INFO graph.BspServiceWorker: startSuperstep: Ready for 
computation on superstep -1 since worker selection and vertex range 
assignments are done in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/-1/_partitionAssignments
12/04/13 09:30:30 INFO graph.BspServiceWorker: reserveInputSplit: 
Reserved input split path /_hadoopBsp/job_local_0001/_inputSplitDir/0, 
overall roughly 0.0% input splits finished
12/04/13 09:30:30 INFO graph.BspServiceWorker: getInputSplitForVertices: 
Reserved /_hadoopBsp/job_local_0001/_inputSplitDir/0 from ZooKeeper and 
got input split ''org.apache.giraph.bsp.BspInputSplit, index=0, num=1'
12/04/13 09:30:30 INFO graph.BspServiceWorker: 
loadVerticesFromInputSplit: Finished loading 
/_hadoopBsp/job_local_0001/_inputSplitDir/0 (v=10, e=20)
12/04/13 09:30:30 INFO graph.BspServiceWorker: reserveInputSplit: 
reservedPath = null, 1 of 1 InputSplits are finished.
12/04/13 09:30:30 INFO graph.BspServiceWorker: setup: Finally loaded a 
total of (v=10, e=20)
12/04/13 09:30:30 INFO graph.BspServiceMaster: barrierOnWorkerList: 1 
out of 1 workers finished on superstep -1 on path 
/_hadoopBsp/job_local_0001/_inputSplitDoneDir
12/04/13 09:30:30 INFO graph.BspService: process: 
inputSplitsAllDoneChanged (all vertices sent from input splits)
12/04/13 09:30:30 INFO graph.BspServiceWorker: moveVerticesToWorker: 
Adding 10 vertices for partition id 0
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: starting for 
superstep -1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 73.0065M
12/04/13 09:30:30 INFO graph.BspServiceMaster: barrierOnWorkerList: 0 
out of 1 workers finished on superstep -1 on path 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerFinishedDir
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: ended for 
superstep -1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 73.00642M
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Superstep -1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 73.00642M
12/04/13 09:30:30 INFO graph.BspServiceMaster: aggregateWorkerStats: 
Aggregation found (vtx=10,finVtx=0,edges=20,msgCount=0) on superstep = -1
12/04/13 09:30:30 INFO graph.BspService: process: superstepFinished signaled
12/04/13 09:30:30 INFO graph.MasterThread: masterThread: Coordination of 
superstep -1 took 0.036 seconds ended with state THIS_SUPERSTEP_DONE and 
is now on superstep 0
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Completed superstep -1 with global stats 
(vtx=10,finVtx=0,edges=20,msgCount=0)
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: prepareSuperstep: 
Superstep 0 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 72.63331M
12/04/13 09:30:30 INFO graph.BspServiceWorker: registerHealth: Created 
my health node for attempt=0, superstep=0 with 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_workerHealthyDir/achingmbp15.local_0

and workerInfo= Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000)
12/04/13 09:30:30 INFO partition.PartitionBalancer: 
balancePartitionsAcrossWorkers: Using algorithm static
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Vertices - Mean: 10, Min: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10, Max: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Edges - Mean: 20, Min: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20, Max: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20
12/04/13 09:30:30 INFO graph.BspService: process: 
partitionAssignmentsReadyChanged (partitions are assigned)
12/04/13 09:30:30 INFO graph.BspServiceWorker: startSuperstep: Ready for 
computation on superstep 0 since worker selection and vertex range 
assignments are done in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_partitionAssignments
12/04/13 09:30:30 INFO graph.GraphMapper: map: Chosen to run ZooKeeper...
12/04/13 09:30:30 INFO graph.BspServiceWorker: sendWorkerPartitions: 
Done sending all my partitions.
12/04/13 09:30:30 INFO graph.BspServiceWorker: exchangeVertexPartitions: 
Done with exchange.
12/04/13 09:30:30 INFO graph.BspServiceWorker: storeCheckpoint: Finished 
metadata 
(_bsp/_checkpoints/job_local_0001/0.achingmbp15.local_0.metadata) and 
vertices (_bsp/_checkpoints/job_local_0001/0.achingmbp15.local_0.vertices).
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: starting for 
superstep 0 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.916M
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: ended for 
superstep 0 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.749374M
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Superstep 0 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.749374M
12/04/13 09:30:30 INFO graph.BspServiceMaster: barrierOnWorkerList: 1 
out of 1 workers finished on superstep 0 on path 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_workerFinishedDir
12/04/13 09:30:30 INFO graph.BspServiceMaster: aggregateWorkerStats: 
Aggregation found (vtx=10,finVtx=0,edges=20,msgCount=20) on superstep = 0
12/04/13 09:30:30 INFO graph.BspService: process: superstepFinished signaled
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Completed superstep 0 with global stats 
(vtx=10,finVtx=0,edges=20,msgCount=20)
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: prepareSuperstep: 
Superstep 1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.749374M
12/04/13 09:30:30 INFO graph.MasterThread: masterThread: Coordination of 
superstep 0 took 0.039 seconds ended with state THIS_SUPERSTEP_DONE and 
is now on superstep 1
12/04/13 09:30:30 INFO graph.BspServiceWorker: registerHealth: Created 
my health node for attempt=0, superstep=1 with 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1/_workerHealthyDir/achingmbp15.local_0

and workerInfo= Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000)
12/04/13 09:30:30 INFO partition.PartitionBalancer: 
balancePartitionsAcrossWorkers: Using algorithm static
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Vertices - Mean: 10, Min: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10, Max: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Edges - Mean: 20, Min: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20, Max: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20
12/04/13 09:30:30 INFO graph.BspService: process: 
partitionAssignmentsReadyChanged (partitions are assigned)
12/04/13 09:30:30 INFO graph.BspServiceMaster: barrierOnWorkerList: 0 
out of 1 workers finished on superstep 1 on path 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1/_workerFinishedDir
12/04/13 09:30:30 INFO graph.BspServiceWorker: startSuperstep: Ready for 
computation on superstep 1 since worker selection and vertex range 
assignments are done in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1/_partitionAssignments
12/04/13 09:30:30 INFO graph.BspServiceWorker: getAggregatorValues: no 
aggregators in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_mergedAggregatorDir

on superstep 1
12/04/13 09:30:30 INFO graph.GraphMapper: map: Chosen to run ZooKeeper...
12/04/13 09:30:30 INFO graph.BspServiceWorker: sendWorkerPartitions: 
Done sending all my partitions.
12/04/13 09:30:30 INFO graph.BspServiceWorker: exchangeVertexPartitions: 
Done with exchange.
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: starting for 
superstep 1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.399864M
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: ended for 
superstep 1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.399864M
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Superstep 1 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.399864M
12/04/13 09:30:30 INFO graph.BspServiceMaster: aggregateWorkerStats: 
Aggregation found (vtx=10,finVtx=0,edges=20,msgCount=20) on superstep = 1
12/04/13 09:30:30 INFO graph.BspService: process: superstepFinished signaled
12/04/13 09:30:30 INFO graph.BspServiceMaster: coordinateSuperstep: 
Cleaning up old Superstep 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Completed superstep 1 with global stats 
(vtx=10,finVtx=0,edges=20,msgCount=20)
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: prepareSuperstep: 
Superstep 2 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 71.2334M
12/04/13 09:30:30 WARN graph.BspService: process: Unknown and 
unprocessed event 
(path=/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_partitionAssignments,

type=NodeDeleted, state=SyncConnected)
12/04/13 09:30:30 INFO graph.BspServiceWorker: registerHealth: Created 
my health node for attempt=0, superstep=2 with 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/2/_workerHealthyDir/achingmbp15.local_0

and workerInfo= Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000)
12/04/13 09:30:30 INFO graph.BspServiceWorker: processEvent : 
partitionExchangeChildrenChanged (at least one worker is done sending 
partitions)
12/04/13 09:30:30 WARN graph.BspService: process: Unknown and 
unprocessed event 
(path=/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_superstepFinished,

type=NodeDeleted, state=SyncConnected)
12/04/13 09:30:30 INFO graph.MasterThread: masterThread: Coordination of 
superstep 1 took 0.039 seconds ended with state THIS_SUPERSTEP_DONE and 
is now on superstep 2
12/04/13 09:30:30 INFO partition.PartitionBalancer: 
balancePartitionsAcrossWorkers: Using algorithm static
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Vertices - Mean: 10, Min: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10, Max: Worker(hostname=achingmbp15.local, 
MRpartition=0, port=30000) - 10
12/04/13 09:30:30 INFO partition.PartitionUtils: analyzePartitionStats: 
Edges - Mean: 20, Min: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20, Max: Worker(hostname=achingmbp15.local, MRpartition=0, 
port=30000) - 20
12/04/13 09:30:30 INFO graph.BspService: process: 
partitionAssignmentsReadyChanged (partitions are assigned)
12/04/13 09:30:30 INFO graph.BspServiceWorker: startSuperstep: Ready for 
computation on superstep 2 since worker selection and vertex range 
assignments are done in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/2/_partitionAssignments
12/04/13 09:30:30 INFO graph.BspServiceWorker: getAggregatorValues: no 
aggregators in 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1/_mergedAggregatorDir

on superstep 2
12/04/13 09:30:30 INFO graph.GraphMapper: map: Chosen to run ZooKeeper...
12/04/13 09:30:30 INFO graph.BspServiceWorker: sendWorkerPartitions: 
Done sending all my partitions.
12/04/13 09:30:30 INFO graph.BspServiceWorker: exchangeVertexPartitions: 
Done with exchange.
12/04/13 09:30:30 INFO graph.BspServiceMaster: barrierOnWorkerList: 0 
out of 1 workers finished on superstep 2 on path 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/2/_workerFinishedDir
12/04/13 09:30:30 INFO graph.BspServiceWorker: storeCheckpoint: Finished 
metadata 
(_bsp/_checkpoints/job_local_0001/2.achingmbp15.local_0.metadata) and 
vertices (_bsp/_checkpoints/job_local_0001/2.achingmbp15.local_0.vertices).
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: starting for 
superstep 2 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 70.699936M
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: flush: ended for 
superstep 2 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 70.699936M
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Superstep 2 totalMem = 81.0625M, maxMem = 123.9375M, freeMem = 70.699936M
12/04/13 09:30:30 INFO graph.BspServiceMaster: aggregateWorkerStats: 
Aggregation found (vtx=10,finVtx=10,edges=20,msgCount=0) on superstep = 2
12/04/13 09:30:30 INFO graph.BspService: process: superstepFinished signaled
12/04/13 09:30:30 INFO graph.BspServiceWorker: finishSuperstep: 
Completed superstep 2 with global stats 
(vtx=10,finVtx=10,edges=20,msgCount=0)
12/04/13 09:30:30 INFO graph.GraphMapper: map: BSP application done 
(global vertices marked done)
12/04/13 09:30:30 INFO graph.GraphMapper: cleanup: Starting for ALL
12/04/13 09:30:30 WARN graph.BspServiceWorker: saveVertices: 
giraph.vertexOutputFormatClass not specified -- there will be no saved 
output
12/04/13 09:30:30 INFO graph.BspServiceMaster: coordinateSuperstep: 
Cleaning up old Superstep 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1
12/04/13 09:30:30 WARN graph.BspService: process: Unknown and 
unprocessed event 
(path=/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1/_partitionAssignments,

type=NodeDeleted, state=SyncConnected)
12/04/13 09:30:30 INFO graph.BspServiceWorker: cleanup: Notifying master 
its okay to cleanup with /_hadoopBsp/job_local_0001/_cleanedUpDir/0_worker
12/04/13 09:30:30 INFO zookeeper.ZooKeeper: Session: 0x136ac8aa05e0001 
closed
12/04/13 09:30:30 INFO zookeeper.ClientCnxn: EventThread shut down
12/04/13 09:30:30 INFO comm.BasicRPCCommunications: close: shutting down 
RPC server
12/04/13 09:30:30 INFO ipc.Server: Stopping server on 30000
12/04/13 09:30:30 WARN graph.BspServiceMaster: coordinateBarrier: 
Already cleaned up 
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/1
12/04/13 09:30:30 INFO ipc.Server: IPC Server handler 0 on 30000: exiting
12/04/13 09:30:30 INFO ipc.Server: Stopping IPC Server listener on 30000
12/04/13 09:30:30 INFO graph.MasterThread: masterThread: Coordination of 
superstep 2 took 0.027 seconds ended with state ALL_SUPERSTEPS_DONE and 
is now on superstep 3
12/04/13 09:30:30 INFO graph.BspServiceMaster: setJobState: 
{"_stateKey":"FINISHED","_applicationAttemptKey":-1,"_superstepKey":-1} 
on superstep 3
12/04/13 09:30:30 INFO metrics.RpcInstrumentation: shut down
12/04/13 09:30:30 INFO ipc.Server: Stopping IPC Server Responder
12/04/13 09:30:30 INFO graph.BspServiceMaster: cleanup: Notifying master 
its okay to cleanup with /_hadoopBsp/job_local_0001/_cleanedUpDir/0_master
12/04/13 09:30:30 INFO graph.BspServiceMaster: cleanUpZooKeeper: Node 
/_hadoopBsp/job_local_0001/_cleanedUpDir already exists, no need to create.
12/04/13 09:30:30 INFO graph.BspServiceMaster: cleanUpZooKeeper: Got 2 
of 2 desired children from /_hadoopBsp/job_local_0001/_cleanedUpDir
12/04/13 09:30:30 INFO graph.BspServiceMaster: cleanupZooKeeper: 
Removing the following path and all children - /_hadoopBsp/job_local_0001
12/04/13 09:30:30 INFO graph.BspService: process: 
masterElectionChildrenChanged signaled
12/04/13 09:30:31 INFO graph.BspService: process: 
cleanedUpChildrenChanged signaled
12/04/13 09:30:31 INFO graph.BspServiceMaster: cleanup: Removed HDFS 
checkpoint directory (_bsp/_checkpoints//job_local_0001) with return = 
true since this job succeeded
12/04/13 09:30:31 INFO zookeeper.ZooKeeper: Session: 0x136ac8aa05e0000 
closed
12/04/13 09:30:31 INFO zookeeper.ClientCnxn: EventThread shut down
12/04/13 09:30:31 INFO graph.MasterThread: setup: Took 0.065 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: vertex input superstep: Took 
0.036 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: superstep 0: Took 0.039 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: superstep 1: Took 0.039 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: superstep 2: Took 0.027 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: shutdown: Took 0.059 seconds.
12/04/13 09:30:31 INFO graph.MasterThread: total: Took 0.266 seconds.
12/04/13 09:30:31 INFO zk.ZooKeeperManager: createZooKeeperClosedStamp: 
Creating my filestamp 
_bsp/_defaultZkManagerDir/job_local_0001/_task/0.COMPUTATION_DONE
12/04/13 09:30:31 INFO zk.ZooKeeperManager: waitUntilAllTasksDone: Got 1 
and 1 desired (polling period is 3000) on attempt 0
12/04/13 09:30:31 INFO zk.ZooKeeperManager: offlineZooKeeperServers: 
waitFor returned 143 and deleted directory 
/Users/aching/git/git_svn_giraph_trunk/_bspZooKeeper
12/04/13 09:30:31 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 
is done. And is in the process of commiting
12/04/13 09:30:33 INFO mapred.LocalJobRunner: finishSuperstep: (all 
workers done) ALL - Attempt=0, Superstep=3
12/04/13 09:30:33 INFO mapred.LocalJobRunner: finishSuperstep: (all 
workers done) ALL - Attempt=0, Superstep=3
12/04/13 09:30:33 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' 
done.
12/04/13 09:30:33 WARN mapred.FileOutputCommitter: Output path is null 
in cleanup


On 4/13/12 4:51 AM, Paolo Castagna wrote:
> Paolo Castagna wrote:
>> This is a better way:
>>
>>    Iterable<String>  results = InternalVertexRunner.run(
>>      SimpleShortestPathsVertex.class,
>>      SimpleShortestPathsVertex.SimpleShortestPathsVertexInputFormat.class,
>>      SimpleShortestPathsVertex.SimpleShortestPathsVertexOutputFormat.class,
>>      params, graph);
>>
>> ... which starts a local ZooKeeper properly.
>>
>> However, I still have a question: when I run it in a unit test everything is
>> fine. When I run it on a Java main method, it hangs towards the end.
> I am using Hadoop 1.0.1, Pig 0.9.2, ZooKeeper 3.4.3 and Giraph from trunk:
>
> [INFO] +- org.apache.hadoop:hadoop-core:jar:1.0.1:compile
> ...
> [INFO] +- org.apache.pig:pig:jar:0.9.2:compile
> ...
> [INFO] +- org.apache.hbase:hbase:jar:0.92.1:compile
> ...
> [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.3:compile
> ...
> [INFO] +- org.apache.giraph:giraph:jar:0.2-SNAPSHOT:compile
>
>
> Paolo



Mime
View raw message