giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikesh Khanna <vik...@stanford.edu>
Subject Re: Zookeeper / Netty (Connection refused) issue for 2 node cluster
Date Wed, 16 Apr 2014 02:13:37 GMT
Hi Tyler, 

To fix "unrecognized -D option", use the following command (just changing the position of
the argument) - 
$HADOOP_HOME/bin/hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado=
op-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dgiraph.zkList=3Dmaster:2181
org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op /user/hduser/output/shortestpaths -w 2 
Thanks, 
Vikesh Khanna, 
Masters, Computer Science (Class of 2015) 
Stanford University 


----- Original Message -----

From: "Tyler Jones" <thomascirca@gmail.com> 
To: user@giraph.apache.org 
Sent: Tuesday, April 15, 2014 7:06:22 PM 
Subject: Zookeeper / Netty (Connection refused) issue for 2 node cluster 

I've set up a two node cluster for Giraph. Currently, I have a master and a
slave node. The /etc/hosts for master looks like:

192.168.1.102 master
192.168.1.152 slave

and for slave, it looks like so:

127.0.0.1       slave
172.0.0.1 slave
192.168.1.102 master
192.168.1.152 slave

I have the following in my core-site.xml on both master and slave.

<property>
<name>giraph.zkList</name>
<value>master:2181</value>
</property>

I currently have Zookeeper running on the master node, on port 2181. I'm
trying to run a simple Giraph job like so:

$HADOOP_HOME/bin/hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado=
op-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/output/shortestpaths -w 2

However, I get the errors as shown below. I'm not sure if its a Zookeeper
or Netty issue. If I do:

telnet master 2181

>From slave and send the ruok string, I get "imok", so I'm not having issues
connecting there. Any thoughts? Originally I thought it was an iptables
issue so I dumped all the rules and the like (iptables was only on slave). I've also tried
removing the
first two entries in /etc/hosts on slave, but that leads to other errors.
Also, if I change the above to:

$HADOOP_HOME/bin/hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado=
op-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation
-Dgiraph.zkList=3Dmaster:2181 -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/output/shortestpaths -w 2

It just says "Unrecognized Option: -D"

Slave

2014-04-15 18:24:07,774 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2014-04-15 18:24:07,901 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2014-04-15 18:24:08,039 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Log level remains at info
2014-04-15 18:24:08,078 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Starting up BspServiceMaster (master thread)...
2014-04-15 18:24:08,096 INFO org.apache.giraph.bsp.BspService:
BspService: Path to create to halt is
/_hadoopBsp/job_201404151217_0002/_haltComputation
2014-04-15 18:24:08,096 INFO org.apache.giraph.bsp.BspService:
BspService: Connecting to ZooKeeper with job job_201404151217_0002, 0
on master:2181
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3D3.4.5-1392090, built on 09/30/2012 17:52
GMT
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=3Dslave
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=3D1.8.0
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=3DOracle Corporation
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=3D/opt/oracle-jdk-bin-1.8.0.0/jre
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=3D/app/hadoop/tmp/mapred/local/taskTracker/hdus=
er/jobcache/job_201404151217_0002/jars/classes:/app/hadoop/tmp/mapred/local=
/taskTracker/hduser/jobcache/job_201404151217_0002/jars:/app/hadoop/tmp/map=
red/local/taskTracker/hduser/jobcache/job_201404151217_0002/attempt_2014041=
51217_0002_m_000000_0/work: / usr/local/hadoop/bin / ../conf:/usr/lib64/jvm/ora=
cle-jdk-bin-1.8/lib/tools.jar: / usr/local/hadoop/bin / ..:/usr/local/hadoop/bi=
n/.. / hadoop-core-0.20.203.0.jar:/usr/local/hadoop/bin / ../lib/aspectjrt-1.6.=
5.jar: / usr/local/hadoop/bin / ../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop= / bin / ..
/ lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/bin / ../lib/common=
s-beanutils-core-1.8.0.jar: / usr/local/hadoop/bin / ../lib/commons-cli-1.2.jar=
: / usr/local/hadoop/bin / .. / lib/commons-codec-1.4.jar:/usr/local/hadoop/bin / .=
. / lib/commons-collections-3.2.1.jar:/usr/local/hadoop/bin / ../lib/commons-co=
nfiguration-1.6.jar: / usr/local/hadoop/bin / ../lib/commons-daemon-1.0.1.jar:/=
usr/local/hadoop/bin/.. / lib/commons-digester-1.8.jar:/usr/local/hadoop/bin / =
.. / lib/commons-el-1.0.jar:/usr/local/hadoop/bin / ../lib/commons-httpclient-3=
.0.1.jar: / usr/local/hadoop/bin / ../lib/commons-lang-2.4.jar:/usr/local/hadoo=
p/bin/.. / lib/commons-logging-1.1.1.jar:/usr/local/hadoop/bin / ../lib/commons=
-logging-api-1.0.4.jar: / usr/local/hadoop/bin / ../lib/commons-math-2.1.jar:/u=
sr/local/hadoop/bin/.. / lib/commons-net-1.4.1.jar:/usr/local/hadoop/bin / ../l=
ib/core-3.1.1.jar: / usr/local/hadoop/bin / ../lib/hsqldb-1.8.0.10.jar:/usr/loc=
al/hadoop/bin/.. / lib/jackson-core-asl-1.0.1.jar:/usr/local/hadoop/bin / ../li=
b/jackson-mapper-asl-1.0.1.jar: / usr/local/hadoop/bin / ../lib/jasper-compiler=
-5.5.12.jar: / usr/local/hadoop/bin / ../lib/jasper-runtime-5.5.12.jar:/usr/loc=
al/hadoop/bin/.. / lib/jets3t-0.6.1.jar:/usr/local/hadoop/bin / ../lib/jetty-6.=
1.26.jar: / usr/local/hadoop/bin / ../lib/jetty-util-6.1.26.jar:/usr/local/hado=
op/bin/.. / lib/jsch-0.1.42.jar:/usr/local/hadoop/bin / ../lib/junit-4.5.jar:/u=
sr/local/hadoop/bin/.. / lib/kfs-0.2.2.jar:/usr/local/hadoop/bin / ../lib/log4j=
-1.2.15.jar: / usr/local/hadoop/bin / ../lib/mockito-all-1.8.5.jar:/usr/local/h=
adoop/bin/.. / lib/oro-2.0.8.jar:/usr/local/hadoop/bin / ../lib/servlet-api-2.5=
-20081211.jar: / usr/local/hadoop/bin / ../lib/slf4j-api-1.4.3.jar:/usr/local/h=
adoop/bin/.. / lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/bin / ../lib/xmlen=
c-0.52.jar: / usr/local/hadoop/bin / ../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hado=
op/bin/../lib/jsp-2.1/jsp-api-2.1.jar
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=3D/usr/local/hadoop/bin/../lib/native/Linux-a=
md64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201404=
151217_0002/attempt_201404151217_0002_m_000000_0/work
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduse=
r/jobcache/job_201404151217_0002/attempt_201404151217_0002_m_000000_0/work/=
tmp
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=3D<NA>
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=3DLinux
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=3Damd64
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=3D3.2.12-gentoo
2014-04-15 18:24:08,102 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=3Dhduser
2014-04-15 18:24:08,105 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=3D/home/hduser
2014-04-15 18:24:08,105 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobc=
ache/job_201404151217_0002/attempt_201404151217_0002_m_000000_0/work
2014-04-15 18:24:08,106 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=3Dmaster:2181
sessionTimeout=3D60000
watcher=3Dorg.apache.giraph.master.BspServiceMaster@3224f60b
2014-04-15 18:24:08,117 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server master/192.168.1.102:2181. Will not
attempt to authenticate using SASL (unknown error)
2014-04-15 18:24:08,118 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to master/192.168.1.102:2181, initiating
session
2014-04-15 18:24:08,141 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server master/192.168.1.102:2181, sessionid
=3D 0x14566270ed7001b, negotiated timeout =3D 40000
2014-04-15 18:24:08,142 INFO org.apache.giraph.bsp.BspService:
process: Asynchronous connection complete.
2014-04-15 18:24:08,147 INFO org.apache.giraph.graph.GraphTaskManager:
map: No need to do anything when not a worker
2014-04-15 18:24:08,148 INFO org.apache.giraph.graph.GraphTaskManager:
cleanup: Starting for MASTER_ONLY
2014-04-15 18:24:08,201 INFO
org.apache.giraph.master.BspServiceMaster: becomeMaster: First child
is '/_hadoopBsp/job_201404151217_0002/_masterElectionDir/slave_00000000000'
and my bid is '/_hadoopBsp/job_201404151217_0002/_masterElectionDir/slave_0=
0000000000'
2014-04-15 18:24:08,246 INFO org.apache.giraph.comm.netty.NettyServer:
NettyServer: Using execution group with 8 threads for
requestFrameDecoder.
2014-04-15 18:24:08,289 INFO org.apache.giraph.comm.netty.NettyServer:
start: Started server communication server: slave/127.0.0.1:30000 with
up to 16 threads on bind attempt 0 with sendBufferSize =3D 32768
receiveBufferSize =3D 524288
2014-04-15 18:24:08,294 INFO org.apache.giraph.comm.netty.NettyClient:
NettyClient: Using execution handler with 8 threads after
request-encoder.
2014-04-15 18:24:08,296 INFO
org.apache.giraph.master.BspServiceMaster: becomeMaster: I am now the
master!
2014-04-15 18:24:08,714 INFO org.apache.giraph.bsp.BspService:
process: applicationAttemptChanged signaled
2014-04-15 18:24:08,790 WARN org.apache.giraph.bsp.BspService:
process: Unknown and unprocessed event
(path=3D/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir/0/_supers=
tepDir,
type=3DNodeChildrenChanged, state=3DSyncConnected)
2014-04-15 18:24:11,321 INFO
org.apache.giraph.io.formats.GiraphFileInputFormat: Total input paths
to process : 1
2014-04-15 18:24:11,328 INFO
org.apache.giraph.master.BspServiceMaster: generateVertexInputSplits:
Got 1 input splits for 2 input threads
2014-04-15 18:24:11,328 WARN
org.apache.giraph.master.BspServiceMaster: createVertexInputSplits:
Number of inputSplits=3D1 < 2=3Dtotal number of input threads, some
threads will be not used
2014-04-15 18:24:11,328 INFO
org.apache.giraph.master.BspServiceMaster: createVertexInputSplits:
Starting to write input split data to zookeeper with 1 threads
2014-04-15 18:24:11,373 INFO
org.apache.giraph.master.BspServiceMaster: createVertexInputSplits:
Done writing input split data to zookeeper
2014-04-15 18:24:11,405 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 18:24:11,409 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 18:24:11,414 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 2 connections, (2 total
connected) 0 failed, 0 failures total.
2014-04-15 18:24:11,415 INFO
org.apache.giraph.partition.PartitionUtils: computePartitionCount:
Creating 4, default would have been 4 partitions.
2014-04-15 18:24:11,835 INFO org.apache.giraph.comm.netty.NettyServer:
start: Using Netty without authentication.
2014-04-15 18:24:11,849 INFO
org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: 0 out
of 2 workers finished on superstep -1 on path
/_hadoopBsp/job_201404151217_0002/_vertexInputSplitDoneDir
2014-04-15 18:24:11,856 INFO
org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList:
Waiting on [master_1, slave_2]


Master:

2014-04-15 12:32:44,937 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library

2014-04-15 12:32:45,082 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2014-04-15 12:32:45,267 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Log level remains at info
2014-04-15 12:32:45,311 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Starting up BspServiceWorker...
2014-04-15 12:32:45,327 INFO org.apache.giraph.bsp.BspService:
BspService: Path to create to halt is
/_hadoopBsp/job_201404151217_0002/_haltComputation
2014-04-15 12:32:45,327 INFO org.apache.giraph.bsp.BspService:
BspService: Connecting to ZooKeeper with job job_201404151217_0002, 1
on master:2181
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3D3.4.5-1392090, built on 09/30/2012 17:52
GMT
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=3Dmaster
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=3D1.7.0_17
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=3DOracle Corporation
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=3D/opt/oracle-jdk-bin-1.7.0.17/jre
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=3D/app/hadoop/tmp/mapred/local/taskTracker/hdus=
er/jobcache/job_201404151217_0002/jars/classes:/app/hadoop/tmp/mapred/local=
/taskTracker/hduser/jobcache/job_201404151217_0002/jars:/app/hadoop/tmp/map=
red/local/taskTracker/hduser/jobcache/job_201404151217_0002/attempt_2014041=
51217_0002_m_000001_0/work: / usr/local/hadoop/bin / ../conf:/usr/lib/jvm/oracl=
e-jdk-bin-1.7/lib/tools.jar: / usr/local/hadoop/bin / ..: / usr/local/hadoop/bin / =
.. / hadoop-core-0.20.203.0.jar:/usr/local/hadoop/bin / ../lib/aspectjrt-1.6.5.=
jar: / usr/local/hadoop/bin / ../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/b=
in/.. / lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/bin / ../lib/commons-=
beanutils-core-1.8.0.jar: / usr/local/hadoop/bin / ../lib/commons-cli-1.2.jar:/=
usr/local/hadoop/bin/.. / lib/commons-codec-1.4.jar:/usr/local/hadoop/bin / ../=
lib/commons-collections-3.2.1.jar: / usr/local/hadoop/bin / ../lib/commons-conf=
iguration-1.6.jar: / usr/local/hadoop/bin / ../lib/commons-daemon-1.0.1.jar:/us=
r/local/hadoop/bin/.. / lib/commons-digester-1.8.jar:/usr/local/hadoop/bin / ..= / lib/commons-el-1.0.jar:/usr/local/hadoop/bin
/ ../lib/commons-httpclient-3.0=
.1.jar: / usr/local/hadoop/bin / .. / lib/commons-lang-2.4.jar:/usr/local/hadoop / =
bin/.. / lib/commons-logging-1.1.1.jar:/usr/local/hadoop/bin / ../lib/commons-l=
ogging-api-1.0.4.jar: / usr/local/hadoop/bin / ../lib/commons-math-2.1.jar:/usr= / local/hadoop/bin
/ .. / lib/commons-net-1.4.1.jar:/usr/local/hadoop/bin / ../lib= / core-3.1.1.jar:/usr/local/hadoop/bin
/ ../lib/hsqldb-1.8.0.10.jar:/usr/local= / hadoop/bin / .. / lib/jackson-core-asl-1.0.1.jar:
/ usr/local/hadoop/bin / ../lib / =
jackson-mapper-asl-1.0.1.jar: / usr/local/hadoop/bin / ../lib/jasper-compiler-5=
.5.12.jar: / usr/local/hadoop/bin / ../lib/jasper-runtime-5.5.12.jar:/usr/local= / hadoop/bin
/ .. / lib/jets3t-0.6.1.jar:/usr/local/hadoop/bin / ../lib/jetty-6.1.=
26.jar: / usr/local/hadoop/bin / ../lib/jetty-util-6.1.26.jar:/usr/local/hadoop= / bin / ..
/ lib/jsch-0.1.42.jar:/usr/local/hadoop/bin / ../lib/junit-4.5.jar:/usr= / local/hadoop/bin
/ .. / lib/kfs-0.2.2.jar:/usr/local/hadoop/bin / ../lib/log4j-1=
.2.15.jar: / usr/local/hadoop/bin / ../lib/mockito-all-1.8.5.jar:/usr/local/had=
oop/bin/.. / lib/oro-2.0.8.jar:/usr/local/hadoop/bin / ../lib/servlet-api-2.5-2=
0081211.jar: / usr/local/hadoop/bin / ../lib/slf4j-api-1.4.3.jar:/usr/local/had=
oop/bin/.. / lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/bin / ../lib/xmlenc-=
0.52.jar: / usr/local/hadoop/bin / ../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop= / bin / ../lib/jsp-2.1/jsp-api-2.1.jar
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=3D/usr/local/hadoop/bin/../lib/native/Linux-a=
md64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201404=
151217_0002/attempt_201404151217_0002_m_000001_0/work
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduse=
r/jobcache/job_201404151217_0002/attempt_201404151217_0002_m_000001_0/work/=
tmp
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=3D<NA>
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=3DLinux
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=3Damd64
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=3D3.4.0-gentoo
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=3Dhduser
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=3D/home/hduser
2014-04-15 12:32:45,334 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobc=
ache/job_201404151217_0002/attempt_201404151217_0002_m_000001_0/work
2014-04-15 12:32:45,335 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=3Dmaster:2181
sessionTimeout=3D60000
watcher=3Dorg.apache.giraph.worker.BspServiceWorker@238cfdf
2014-04-15 12:32:45,349 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server master/192.168.1.102:2181. Will not
attempt to authenticate using SASL (unknown error)
2014-04-15 12:32:45,349 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to master/192.168.1.102:2181, initiating
session
2014-04-15 12:32:45,359 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server master/192.168.1.102:2181, sessionid
=3D 0x14566270ed7001c, negotiated timeout =3D 40000
2014-04-15 12:32:45,361 INFO org.apache.giraph.bsp.BspService:
process: Asynchronous connection complete.
2014-04-15 12:32:45,424 INFO org.apache.giraph.comm.netty.NettyServer:
NettyServer: Using execution group with 8 threads for
requestFrameDecoder.
2014-04-15 12:32:45,477 INFO org.apache.giraph.comm.netty.NettyServer:
start: Started server communication server: master/192.168.1.102:30001
with up to 16 threads on bind attempt 0 with sendBufferSize =3D 32768
receiveBufferSize =3D 524288
2014-04-15 12:32:45,481 INFO org.apache.giraph.comm.netty.NettyClient:
NettyClient: Using execution handler with 8 threads after
request-encoder.
2014-04-15 12:32:45,489 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Registering health of this worker...
2014-04-15 12:32:45,504 INFO org.apache.giraph.bsp.BspService:
getJobState: Job state already exists
(/_hadoopBsp/job_201404151217_0002/_masterJobState)
2014-04-15 12:32:45,524 INFO org.apache.giraph.bsp.BspService:
getApplicationAttempt: Node
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir already
exists!
2014-04-15 12:32:45,549 INFO org.apache.giraph.bsp.BspService:
getApplicationAttempt: Node
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir already
exists!
2014-04-15 12:32:45,566 INFO
org.apache.giraph.worker.BspServiceWorker: registerHealth: Created my
health node for attempt=3D0, superstep=3D-1 with
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir/0/_superstepDir/-=
1/_workerHealthyDir/master_1
and workerInfo=3D Worker(hostname=3Dmaster, MRtaskID=3D1, port=3D30001)
2014-04-15 12:32:45,983 INFO org.apache.giraph.comm.netty.NettyServer:
start: Using Netty without authentication.
2014-04-15 12:32:46,389 INFO org.apache.giraph.bsp.BspService:
process: partitionAssignmentsReadyChanged (partitions are assigned)
2014-04-15 12:32:46,393 INFO
org.apache.giraph.worker.BspServiceWorker: startSuperstep:
Master(hostname=3Dslave, MRtaskID=3D0, port=3D30000)
2014-04-15 12:32:46,393 INFO
org.apache.giraph.worker.BspServiceWorker: startSuperstep: Ready for
computation on superstep -1 since worker selection and vertex range
assignments are done in
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir/0/_superstepDir/-=
1/_addressesAndPartitions
2014-04-15 12:32:46,395 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,396 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,398 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 0 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,398 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 1 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,398 INFO org.apache.giraph.comm.netty.NettyServer:
start: Using Netty without authentication.
2014-04-15 12:32:46,398 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,399 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 2 failures total.
2014-04-15 12:32:46,399 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,399 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 2 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,399 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 3 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,400 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 4 failures total.
2014-04-15 12:32:46,402 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,402 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,403 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 4 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,403 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 5 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,403 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,403 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 6 failures total.
2014-04-15 12:32:46,404 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,404 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 6 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,404 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 7 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,404 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,404 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 8 failures total.
2014-04-15 12:32:46,404 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,404 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 8 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,405 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,405 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 9 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,405 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 10 failures total.
2014-04-15 12:32:46,405 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,405 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 10 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,405 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 11 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,405 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,406 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 12 failures total.
2014-04-15 12:32:46,406 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,406 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 12 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,406 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,406 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 13 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,407 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 14 failures total.
2014-04-15 12:32:46,407 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,407 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30002 with 14 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30002
2014-04-15 12:32:46,407 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 12:32:46,408 WARN org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Future failed to connect with
slave/192.168.1.152:30000 with 15 failures because of
java.net.ConnectException: Connection refused:
slave/192.168.1.152:30000
2014-04-15 12:32:46,408 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total
connected) 2 failed, 16 failures total.
2014-04-15 12:32:46,408 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.


Slave:

2014-04-15 18:24:10,782 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2014-04-15 18:24:10,907 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2014-04-15 18:24:11,047 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Log level remains at info
2014-04-15 18:24:11,085 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Starting up BspServiceWorker...
2014-04-15 18:24:11,103 INFO org.apache.giraph.bsp.BspService:
BspService: Path to create to halt is
/_hadoopBsp/job_201404151217_0002/_haltComputation
2014-04-15 18:24:11,103 INFO org.apache.giraph.bsp.BspService:
BspService: Connecting to ZooKeeper with job job_201404151217_0002, 2
on master:2181
2014-04-15 18:24:11,109 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3D3.4.5-1392090, built on 09/30/2012 17:52
GMT
2014-04-15 18:24:11,109 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=3Dslave
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=3D1.8.0
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=3DOracle Corporation
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=3D/opt/oracle-jdk-bin-1.8.0.0/jre
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=3D/app/hadoop/tmp/mapred/local/taskTracker/hdus=
er/jobcache/job_201404151217_0002/jars/classes:/app/hadoop/tmp/mapred/local=
/taskTracker/hduser/jobcache/job_201404151217_0002/jars:/app/hadoop/tmp/map=
red/local/taskTracker/hduser/jobcache/job_201404151217_0002/attempt_2014041=
51217_0002_m_000002_0/work: / usr/local/hadoop/bin / ../conf:/usr/lib64/jvm/ora=
cle-jdk-bin-1.8/lib/tools.jar: / usr/local/hadoop/bin / ..:/usr/local/hadoop/bi=
n/.. / hadoop-core-0.20.203.0.jar:/usr/local/hadoop/bin / ../lib/aspectjrt-1.6.=
5.jar: / usr/local/hadoop/bin / ../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop= / bin / ..
/ lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/bin / ../lib/common=
s-beanutils-core-1.8.0.jar: / usr/local/hadoop/bin / ../lib/commons-cli-1.2.jar=
: / usr/local/hadoop/bin / .. / lib/commons-codec-1.4.jar:/usr/local/hadoop/bin / .=
. / lib/commons-collections-3.2.1.jar:/usr/local/hadoop/bin / ../lib/commons-co=
nfiguration-1.6.jar: / usr/local/hadoop/bin / ../lib/commons-daemon-1.0.1.jar:/=
usr/local/hadoop/bin/.. / lib/commons-digester-1.8.jar:/usr/local/hadoop/bin / =
.. / lib/commons-el-1.0.jar:/usr/local/hadoop/bin / ../lib/commons-httpclient-3=
.0.1.jar: / usr/local/hadoop/bin / ../lib/commons-lang-2.4.jar:/usr/local/hadoo=
p/bin/.. / lib/commons-logging-1.1.1.jar:/usr/local/hadoop/bin / ../lib/commons=
-logging-api-1.0.4.jar: / usr/local/hadoop/bin / ../lib/commons-math-2.1.jar:/u=
sr/local/hadoop/bin/.. / lib/commons-net-1.4.1.jar:/usr/local/hadoop/bin / ../l=
ib/core-3.1.1.jar: / usr/local/hadoop/bin / ../lib/hsqldb-1.8.0.10.jar:/usr/loc=
al/hadoop/bin/.. / lib/jackson-core-asl-1.0.1.jar:/usr/local/hadoop/bin / ../li=
b/jackson-mapper-asl-1.0.1.jar: / usr/local/hadoop/bin / ../lib/jasper-compiler=
-5.5.12.jar: / usr/local/hadoop/bin / ../lib/jasper-runtime-5.5.12.jar:/usr/loc=
al/hadoop/bin/.. / lib/jets3t-0.6.1.jar:/usr/local/hadoop/bin / ../lib/jetty-6.=
1.26.jar: / usr/local/hadoop/bin / ../lib/jetty-util-6.1.26.jar:/usr/local/hado=
op/bin/.. / lib/jsch-0.1.42.jar:/usr/local/hadoop/bin / ../lib/junit-4.5.jar:/u=
sr/local/hadoop/bin/.. / lib/kfs-0.2.2.jar:/usr/local/hadoop/bin / ../lib/log4j=
-1.2.15.jar: / usr/local/hadoop/bin / ../lib/mockito-all-1.8.5.jar:/usr/local/h=
adoop/bin/.. / lib/oro-2.0.8.jar:/usr/local/hadoop/bin / ../lib/servlet-api-2.5=
-20081211.jar: / usr/local/hadoop/bin / ../lib/slf4j-api-1.4.3.jar:/usr/local/h=
adoop/bin/.. / lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/bin / ../lib/xmlen=
c-0.52.jar: / usr/local/hadoop/bin / ../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hado=
op/bin/../lib/jsp-2.1/jsp-api-2.1.jar
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=3D/usr/local/hadoop/bin/../lib/native/Linux-a=
md64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201404=
151217_0002/attempt_201404151217_0002_m_000002_0/work
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduse=
r/jobcache/job_201404151217_0002/attempt_201404151217_0002_m_000002_0/work/=
tmp
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=3D<NA>
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=3DLinux
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=3Damd64
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=3D3.2.12-gentoo
2014-04-15 18:24:11,110 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=3Dhduser
2014-04-15 18:24:11,116 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=3D/home/hduser
2014-04-15 18:24:11,116 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=3D/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobc=
ache/job_201404151217_0002/attempt_201404151217_0002_m_000002_0/work
2014-04-15 18:24:11,117 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=3Dmaster:2181
sessionTimeout=3D60000
watcher=3Dorg.apache.giraph.worker.BspServiceWorker@3224f60b
2014-04-15 18:24:11,128 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server master/192.168.1.102:2181. Will not
attempt to authenticate using SASL (unknown error)
2014-04-15 18:24:11,129 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to master/192.168.1.102:2181, initiating
session
2014-04-15 18:24:11,141 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server master/192.168.1.102:2181, sessionid
=3D 0x14566270ed7001d, negotiated timeout =3D 40000
2014-04-15 18:24:11,142 INFO org.apache.giraph.bsp.BspService:
process: Asynchronous connection complete.
2014-04-15 18:24:11,185 INFO org.apache.giraph.comm.netty.NettyServer:
NettyServer: Using execution group with 8 threads for
requestFrameDecoder.
2014-04-15 18:24:11,229 INFO org.apache.giraph.comm.netty.NettyServer:
start: Started server communication server: slave/127.0.0.1:30002 with
up to 16 threads on bind attempt 0 with sendBufferSize =3D 32768
receiveBufferSize =3D 524288
2014-04-15 18:24:11,233 INFO org.apache.giraph.comm.netty.NettyClient:
NettyClient: Using execution handler with 8 threads after
request-encoder.
2014-04-15 18:24:11,240 INFO org.apache.giraph.graph.GraphTaskManager:
setup: Registering health of this worker...
2014-04-15 18:24:11,252 INFO org.apache.giraph.bsp.BspService:
getJobState: Job state already exists
(/_hadoopBsp/job_201404151217_0002/_masterJobState)
2014-04-15 18:24:11,264 INFO org.apache.giraph.bsp.BspService:
getApplicationAttempt: Node
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir already
exists!
2014-04-15 18:24:11,273 INFO org.apache.giraph.bsp.BspService:
getApplicationAttempt: Node
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir already
exists!
2014-04-15 18:24:11,282 INFO
org.apache.giraph.worker.BspServiceWorker: registerHealth: Created my
health node for attempt=3D0, superstep=3D-1 with
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir/0/_superstepDir/-=
1/_workerHealthyDir/slave_2
and workerInfo=3D Worker(hostname=3Dslave, MRtaskID=3D2, port=3D30002)
2014-04-15 18:24:11,415 INFO org.apache.giraph.comm.netty.NettyServer:
start: Using Netty without authentication.
2014-04-15 18:24:11,821 INFO org.apache.giraph.bsp.BspService:
process: partitionAssignmentsReadyChanged (partitions are assigned)
2014-04-15 18:24:11,826 INFO
org.apache.giraph.worker.BspServiceWorker: startSuperstep:
Master(hostname=3Dslave, MRtaskID=3D0, port=3D30000)
2014-04-15 18:24:11,826 INFO
org.apache.giraph.worker.BspServiceWorker: startSuperstep: Ready for
computation on superstep -1 since worker selection and vertex range
assignments are done in
/_hadoopBsp/job_201404151217_0002/_applicationAttemptsDir/0/_superstepDir/-=
1/_addressesAndPartitions
2014-04-15 18:24:11,828 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 18:24:11,829 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2014-04-15 18:24:11,834 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 2 connections, (2 total
connected) 0 failed, 0 failures total.
2014-04-15 18:24:11,848 INFO
org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server
window metrics MBytes/sec received =3D 0, MBytesReceived =3D 0.0001, ave
received req MBytes =3D 0.0001, secs waited =3D 1.39760064E9
2014-04-15 18:24:21,859 INFO
org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits:
Waiting for 1 more tasks to send their aggregator data, task ids: [1]
2014-04-15 18:24:41,860 INFO
org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits:
Waiting for 1 more tasks to send their aggregator data, task ids: [1]
2014-04-15 18:25:01,860 INFO
org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits:
Waiting for 1 more tasks to send their aggregator data, task ids: [1]
2014-04-15 18:25:21,860 INFO
org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits:
Waiting for 1 more tasks to send their aggregator data, task ids: [1]
2014-04-15 18:25:31,861 INFO
org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits:
Waiting for 1 more tasks to send their aggregator data, task ids: [1] 


Mime
View raw message