hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuesheng Hu <yueshen...@gmail.com>
Subject large input on k-means failed
Date Sat, 15 Sep 2012 07:53:34 GMT
Hi,

     I am testing k-means' performance on large data. I generated 5 bilion
vectors via mapreduce(about 140GB),  and launched 45 tasks to handle the 45
input files in the input directory. Every taks need to process about 3GB
data. But the job failed when the iteration ends.  The following is log
message:

attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO
zookeeper.ZooKeeper: Client environment:os.version=3.3.4-5.fc17.x86_64
attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO
zookeeper.ZooKeeper: Client environment:user.name=hadoop
attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO
zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO
zookeeper.ZooKeeper: Client
environment:user.dir=/tmp/hadoop-hadoop/bsp/local/groomServer/attempt_201209141233_0001_000001_0/work
attempt_201209141233_0001_000001_0: 12/09/14 12:35:40 INFO
zookeeper.ZooKeeper: Initiating client connection, connectString=
192.168.1.200:21810 sessionTimeout=1200000
watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@76dfa2ae
attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO
zookeeper.ClientCnxn: Opening socket connection to server /
192.168.1.200:21810
attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO
zookeeper.ClientCnxn: Socket connection established to namenode/
192.168.1.200:21810, initiating session
attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO
sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At datanode01/
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:35:41 INFO
zookeeper.ClientCnxn: Session establishment complete on server namenode/
192.168.1.200:21810, sessionid = 0x139c30dba450011, negotiated timeout =
1200000
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x602cc296, /192.168.1.211:56957 => /192.168.1.211:61002] CONNECTED: /
192.168.1.211:56957
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30a945f8, /192.168.1.216:56518 => /192.168.1.211:61002] CONNECTED: /
192.168.1.216:56518
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7ca7700a, /192.168.1.218:39111 => /192.168.1.211:61002] CONNECTED: /
192.168.1.218:39111
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3930b5bd, /192.168.1.216:56519 => /192.168.1.211:61002] CONNECTED: /
192.168.1.216:56519
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x5a716968, /192.168.1.217:46340 => /192.168.1.211:61002] CONNECTED: /
192.168.1.217:46340
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2ceba8a0, /192.168.1.212:56842 => /192.168.1.211:61002] CONNECTED: /
192.168.1.212:56842
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x626bb08a, /192.168.1.219:50114 => /192.168.1.211:61002] CONNECTED: /
192.168.1.219:50114
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c136681, /192.168.1.219:50115 => /192.168.1.211:61002] CONNECTED: /
192.168.1.219:50115
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x694043b3, /192.168.1.217:46341 => /192.168.1.211:61002] CONNECTED: /
192.168.1.217:46341
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7a1cd2c0, /192.168.1.214:39405 => /192.168.1.211:61002] CONNECTED: /
192.168.1.214:39405
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6319a8b7, /192.168.1.216:56521 => /192.168.1.211:61002] CONNECTED: /
192.168.1.216:56521
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7d083fe3, /192.168.1.218:39112 => /192.168.1.211:61002] CONNECTED: /
192.168.1.218:39112
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x238bb76a, /192.168.1.212:56843 => /192.168.1.211:61002] CONNECTED: /
192.168.1.212:56843
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1cbbbc31, /192.168.1.215:35166 => /192.168.1.211:61002] CONNECTED: /
192.168.1.215:35166
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x04a7b933, /192.168.1.214:39406 => /192.168.1.211:61002] CONNECTED: /
192.168.1.214:39406
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0c7b7e61, /192.168.1.218:39113 => /192.168.1.211:61002] CONNECTED: /
192.168.1.218:39113
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x32e796ac] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x32e796ac, /192.168.1.211:56960 => datanode01/
192.168.1.211:61002] BOUND: /192.168.1.211:56960
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x32e796ac, /192.168.1.211:56960 => datanode01/
192.168.1.211:61002] CONNECTED: datanode01/192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x74a77044, /192.168.1.215:35165 => /192.168.1.211:61002] CONNECTED: /
192.168.1.215:35165
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x638c7ceb, /192.168.1.216:56520 => /192.168.1.211:61002] CONNECTED: /
192.168.1.216:56520
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7f7d0638, /192.168.1.218:39114 => /192.168.1.211:61002] CONNECTED: /
192.168.1.218:39114
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x73b034c8, /192.168.1.218:39115 => /192.168.1.211:61002] CONNECTED: /
192.168.1.218:39115
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1c8d542b, /192.168.1.212:56845 => /192.168.1.211:61002] CONNECTED: /
192.168.1.212:56845
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x2b5cf2a8, /192.168.1.217:46344 => /192.168.1.211:61002] CONNECTED: /
192.168.1.217:46344
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x351672ae, /192.168.1.211:56959 => /192.168.1.211:61002] CONNECTED: /
192.168.1.211:56959
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x743cfb45, /192.168.1.211:56961 => /192.168.1.211:61002] CONNECTED: /
192.168.1.211:56961
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1f6fc5a8, /192.168.1.212:56844 => /192.168.1.211:61002] CONNECTED: /
192.168.1.212:56844
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x4e5cd46b, /192.168.1.214:39407 => /192.168.1.211:61002] CONNECTED: /
192.168.1.214:39407
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x7fa618ad, /192.168.1.217:46342 => /192.168.1.211:61002] CONNECTED: /
192.168.1.217:46342
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1bb08b3a, /192.168.1.213:50617 => /192.168.1.211:61002] CONNECTED: /
192.168.1.213:50617
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x63849b09, /192.168.1.213:50618 => /192.168.1.211:61002] CONNECTED: /
192.168.1.213:50618
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x44d5ce53, /192.168.1.217:46343 => /192.168.1.211:61002] CONNECTED: /
192.168.1.217:46343
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x0290e1b2, /192.168.1.216:56522 => /192.168.1.211:61002] CONNECTED: /
192.168.1.216:56522
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x30dcce73, /192.168.1.219:50116 => /192.168.1.211:61002] CONNECTED: /
192.168.1.219:50116
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75d16e6e, /192.168.1.219:50118 => /192.168.1.211:61002] CONNECTED: /
192.168.1.219:50118
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x02551050, /192.168.1.215:35168 => /192.168.1.211:61002] CONNECTED: /
192.168.1.215:35168
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6bee8942, /192.168.1.211:56960 => /192.168.1.211:61002] CONNECTED: /
192.168.1.211:56960
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x633f09d5, /192.168.1.212:56846 => /192.168.1.211:61002] CONNECTED: /
192.168.1.212:56846
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x1ca2d8f6, /192.168.1.213:50619 => /192.168.1.211:61002] CONNECTED: /
192.168.1.213:50619
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x69b24aa6, /192.168.1.214:39409 => /192.168.1.211:61002] CONNECTED: /
192.168.1.214:39409
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x21645be6, /192.168.1.213:50616 => /192.168.1.211:61002] CONNECTED: /
192.168.1.213:50616
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x65371c10, /192.168.1.211:56958 => /192.168.1.211:61002] CONNECTED: /
192.168.1.211:56958
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x16de9eea, /192.168.1.214:39408 => /192.168.1.211:61002] CONNECTED: /
192.168.1.214:39408
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x6dbf1737, /192.168.1.219:50117 => /192.168.1.211:61002] CONNECTED: /
192.168.1.219:50117
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x669b0d36, /192.168.1.215:35167 => /192.168.1.211:61002] CONNECTED: /
192.168.1.215:35167
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x75304935, /192.168.1.215:35169 => /192.168.1.211:61002] CONNECTED: /
192.168.1.215:35169
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] BOUND: /
192.168.1.211:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO ipc.NettyServer:
[id: 0x3f801e2c, /192.168.1.213:50620 => /192.168.1.211:61002] CONNECTED: /
192.168.1.213:50620
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x693e6031] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x693e6031, /192.168.1.211:38151 => datanode02/
192.168.1.212:61001] BOUND: /192.168.1.211:38151
attempt_201209141233_0001_000001_0: 12/09/14 12:49:01 INFO
ipc.NettyTransceiver: [id: 0x693e6031, /192.168.1.211:38151 => datanode02/
192.168.1.212:61001] CONNECTED: datanode02/192.168.1.212:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x4d506e73] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x4d506e73, /192.168.1.211:53194 => datanode01/
192.168.1.211:61001] BOUND: /192.168.1.211:53194
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x4d506e73, /192.168.1.211:53194 => datanode01/
192.168.1.211:61001] CONNECTED: datanode01/192.168.1.211:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x122e8a3c] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x122e8a3c, /192.168.1.211:40394 => datanode02/
192.168.1.212:61003] BOUND: /192.168.1.211:40394
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x122e8a3c, /192.168.1.211:40394 => datanode02/
192.168.1.212:61003] CONNECTED: datanode02/192.168.1.212:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6eb54678] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6eb54678, /192.168.1.211:56565 => datanode03/
192.168.1.213:61002] BOUND: /192.168.1.211:56565
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6eb54678, /192.168.1.211:56565 => datanode03/
192.168.1.213:61002] CONNECTED: datanode03/192.168.1.213:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x48b565d2] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x48b565d2, /192.168.1.211:53664 => datanode01/
192.168.1.211:61004] BOUND: /192.168.1.211:53664
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x48b565d2, /192.168.1.211:53664 => datanode01/
192.168.1.211:61004] CONNECTED: datanode01/192.168.1.211:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x1b7d8bfa] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x1b7d8bfa, /192.168.1.211:36872 => datanode04/
192.168.1.214:61001] BOUND: /192.168.1.211:36872
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x1b7d8bfa, /192.168.1.211:36872 => datanode04/
192.168.1.214:61001] CONNECTED: datanode04/192.168.1.214:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x41424f86] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x41424f86, /192.168.1.211:39778 => datanode02/
192.168.1.212:61002] BOUND: /192.168.1.211:39778
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x41424f86, /192.168.1.211:39778 => datanode02/
192.168.1.212:61002] CONNECTED: datanode02/192.168.1.212:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x53b252a4] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x53b252a4, /192.168.1.211:45705 => datanode03/
192.168.1.213:61001] BOUND: /192.168.1.211:45705
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x53b252a4, /192.168.1.211:45705 => datanode03/
192.168.1.213:61001] CONNECTED: datanode03/192.168.1.213:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x0be83fed] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x0be83fed, /192.168.1.211:38537 => datanode01/
192.168.1.211:61003] BOUND: /192.168.1.211:38537
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x0be83fed, /192.168.1.211:38537 => datanode01/
192.168.1.211:61003] CONNECTED: datanode01/192.168.1.211:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x45697ab0] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x45697ab0, /192.168.1.211:39655 => datanode07/
192.168.1.217:61001] BOUND: /192.168.1.211:39655
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x45697ab0, /192.168.1.211:39655 => datanode07/
192.168.1.217:61001] CONNECTED: datanode07/192.168.1.217:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2f555d7c] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2f555d7c, /192.168.1.211:59371 => datanode06/
192.168.1.216:61002] BOUND: /192.168.1.211:59371
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2f555d7c, /192.168.1.211:59371 => datanode06/
192.168.1.216:61002] CONNECTED: datanode06/192.168.1.216:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x78634aff] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x78634aff, /192.168.1.211:53018 => datanode03/
192.168.1.213:61005] BOUND: /192.168.1.211:53018
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x78634aff, /192.168.1.211:53018 => datanode03/
192.168.1.213:61005] CONNECTED: datanode03/192.168.1.213:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x79b35d3a] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x79b35d3a, /192.168.1.211:58515 => datanode04/
192.168.1.214:61004] BOUND: /192.168.1.211:58515
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x79b35d3a, /192.168.1.211:58515 => datanode04/
192.168.1.214:61004] CONNECTED: datanode04/192.168.1.214:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x3c90cbce] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x3c90cbce, /192.168.1.211:37935 => datanode05/
192.168.1.215:61003] BOUND: /192.168.1.211:37935
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x3c90cbce, /192.168.1.211:37935 => datanode05/
192.168.1.215:61003] CONNECTED: datanode05/192.168.1.215:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2c1768fe] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2c1768fe, /192.168.1.211:60504 => datanode08/
192.168.1.218:61001] BOUND: /192.168.1.211:60504
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x2c1768fe, /192.168.1.211:60504 => datanode08/
192.168.1.218:61001] CONNECTED: datanode08/192.168.1.218:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x240e7423] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x240e7423, /192.168.1.211:39924 => datanode07/
192.168.1.217:61002] BOUND: /192.168.1.211:39924
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x240e7423, /192.168.1.211:39924 => datanode07/
192.168.1.217:61002] CONNECTED: datanode07/192.168.1.217:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x75302bb6] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x75302bb6, /192.168.1.211:47363 => datanode06/
192.168.1.216:61003] BOUND: /192.168.1.211:47363
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x75302bb6, /192.168.1.211:47363 => datanode06/
192.168.1.216:61003] CONNECTED: datanode06/192.168.1.216:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6392c99c] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6392c99c, /192.168.1.211:50035 => datanode04/
192.168.1.214:61005] BOUND: /192.168.1.211:50035
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x6392c99c, /192.168.1.211:50035 => datanode04/
192.168.1.214:61005] CONNECTED: datanode04/192.168.1.214:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x798623a3] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x798623a3, /192.168.1.211:53262 => datanode05/
192.168.1.215:61004] BOUND: /192.168.1.211:53262
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x798623a3, /192.168.1.211:53262 => datanode05/
192.168.1.215:61004] CONNECTED: datanode05/192.168.1.215:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x13b71ee8] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x13b71ee8, /192.168.1.211:57211 => datanode03/
192.168.1.213:61003] BOUND: /192.168.1.211:57211
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x13b71ee8, /192.168.1.211:57211 => datanode03/
192.168.1.213:61003] CONNECTED: datanode03/192.168.1.213:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: Connecting to datanode01/192.168.1.211:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:02 INFO
ipc.NettyTransceiver: [id: 0x4f08370b] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4f08370b, /192.168.1.211:41614 => datanode01/
192.168.1.211:61005] BOUND: /192.168.1.211:41614
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4f08370b, /192.168.1.211:41614 => datanode01/
192.168.1.211:61005] CONNECTED: datanode01/192.168.1.211:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x57e13166] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x57e13166, /192.168.1.211:48980 => datanode02/
192.168.1.212:61004] BOUND: /192.168.1.211:48980
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x57e13166, /192.168.1.211:48980 => datanode02/
192.168.1.212:61004] CONNECTED: datanode02/192.168.1.212:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1aa0c35b] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1aa0c35b, /192.168.1.211:56959 => datanode04/
192.168.1.214:61002] BOUND: /192.168.1.211:56959
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1aa0c35b, /192.168.1.211:56959 => datanode04/
192.168.1.214:61002] CONNECTED: datanode04/192.168.1.214:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x484eff79] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x484eff79, /192.168.1.211:50664 => datanode05/
192.168.1.215:61001] BOUND: /192.168.1.211:50664
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x484eff79, /192.168.1.211:50664 => datanode05/
192.168.1.215:61001] CONNECTED: datanode05/192.168.1.215:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x10bd314a] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x10bd314a, /192.168.1.211:40716 => datanode06/
192.168.1.216:61001] BOUND: /192.168.1.211:40716
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x10bd314a, /192.168.1.211:40716 => datanode06/
192.168.1.216:61001] CONNECTED: datanode06/192.168.1.216:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode02/192.168.1.212:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x5b12bba0] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x5b12bba0, /192.168.1.211:50285 => datanode02/
192.168.1.212:61005] BOUND: /192.168.1.211:50285
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x5b12bba0, /192.168.1.211:50285 => datanode02/
192.168.1.212:61005] CONNECTED: datanode02/192.168.1.212:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode03/192.168.1.213:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x7659657a] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x7659657a, /192.168.1.211:44243 => datanode03/
192.168.1.213:61004] BOUND: /192.168.1.211:44243
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x7659657a, /192.168.1.211:44243 => datanode03/
192.168.1.213:61004] CONNECTED: datanode03/192.168.1.213:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode04/192.168.1.214:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x64fed7fa] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x64fed7fa, /192.168.1.211:37357 => datanode04/
192.168.1.214:61003] BOUND: /192.168.1.211:37357
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x64fed7fa, /192.168.1.211:37357 => datanode04/
192.168.1.214:61003] CONNECTED: datanode04/192.168.1.214:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x500dffbf] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x500dffbf, /192.168.1.211:36378 => datanode05/
192.168.1.215:61002] BOUND: /192.168.1.211:36378
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x500dffbf, /192.168.1.211:36378 => datanode05/
192.168.1.215:61002] CONNECTED: datanode05/192.168.1.215:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4861c19c] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4861c19c, /192.168.1.211:47948 => datanode09/
192.168.1.219:61003] BOUND: /192.168.1.211:47948
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4861c19c, /192.168.1.211:47948 => datanode09/
192.168.1.219:61003] CONNECTED: datanode09/192.168.1.219:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3a67ba87] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3a67ba87, /192.168.1.211:52403 => datanode08/
192.168.1.218:61004] BOUND: /192.168.1.211:52403
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3a67ba87, /192.168.1.211:52403 => datanode08/
192.168.1.218:61004] CONNECTED: datanode08/192.168.1.218:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x389f2c6e] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x389f2c6e, /192.168.1.211:43475 => datanode07/
192.168.1.217:61005] BOUND: /192.168.1.211:43475
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x389f2c6e, /192.168.1.211:43475 => datanode07/
192.168.1.217:61005] CONNECTED: datanode07/192.168.1.217:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x6786091b] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x6786091b, /192.168.1.211:53587 => datanode09/
192.168.1.219:61004] BOUND: /192.168.1.211:53587
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x6786091b, /192.168.1.211:53587 => datanode09/
192.168.1.219:61004] CONNECTED: datanode09/192.168.1.219:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x02cb7647] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x02cb7647, /192.168.1.211:47875 => datanode08/
192.168.1.218:61005] BOUND: /192.168.1.211:47875
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x02cb7647, /192.168.1.211:47875 => datanode08/
192.168.1.218:61005] CONNECTED: datanode08/192.168.1.218:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4158fa62] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4158fa62, /192.168.1.211:38157 => datanode09/
192.168.1.219:61001] BOUND: /192.168.1.211:38157
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x4158fa62, /192.168.1.211:38157 => datanode09/
192.168.1.219:61001] CONNECTED: datanode09/192.168.1.219:61001
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3ed62895] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3ed62895, /192.168.1.211:60073 => datanode08/
192.168.1.218:61002] BOUND: /192.168.1.211:60073
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x3ed62895, /192.168.1.211:60073 => datanode08/
192.168.1.218:61002] CONNECTED: datanode08/192.168.1.218:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x12977ada] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x12977ada, /192.168.1.211:48821 => datanode07/
192.168.1.217:61003] BOUND: /192.168.1.211:48821
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x12977ada, /192.168.1.211:48821 => datanode07/
192.168.1.217:61003] CONNECTED: datanode07/192.168.1.217:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x42e95cbf] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x42e95cbf, /192.168.1.211:36170 => datanode06/
192.168.1.216:61004] BOUND: /192.168.1.211:36170
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x42e95cbf, /192.168.1.211:36170 => datanode06/
192.168.1.216:61004] CONNECTED: datanode06/192.168.1.216:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode05/192.168.1.215:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x37e66f47] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x37e66f47, /192.168.1.211:43133 => datanode05/
192.168.1.215:61005] BOUND: /192.168.1.211:43133
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x37e66f47, /192.168.1.211:43133 => datanode05/
192.168.1.215:61005] CONNECTED: datanode05/192.168.1.215:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x69bf5fb1] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x69bf5fb1, /192.168.1.211:39261 => datanode09/
192.168.1.219:61002] BOUND: /192.168.1.211:39261
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x69bf5fb1, /192.168.1.211:39261 => datanode09/
192.168.1.219:61002] CONNECTED: datanode09/192.168.1.219:61002
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode08/192.168.1.218:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x36d0c62b] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x36d0c62b, /192.168.1.211:55267 => datanode08/
192.168.1.218:61003] BOUND: /192.168.1.211:55267
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x36d0c62b, /192.168.1.211:55267 => datanode08/
192.168.1.218:61003] CONNECTED: datanode08/192.168.1.218:61003
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode07/192.168.1.217:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1bac8b63] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1bac8b63, /192.168.1.211:48745 => datanode07/
192.168.1.217:61004] BOUND: /192.168.1.211:48745
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1bac8b63, /192.168.1.211:48745 => datanode07/
192.168.1.217:61004] CONNECTED: datanode07/192.168.1.217:61004
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: Connecting to datanode06/192.168.1.216:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1e59a7ba] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1e59a7ba, /192.168.1.211:39669 => datanode06/
192.168.1.216:61005] BOUND: /192.168.1.211:39669
attempt_201209141233_0001_000001_0: 12/09/14 12:49:03 INFO
ipc.NettyTransceiver: [id: 0x1e59a7ba, /192.168.1.211:39669 => datanode06/
192.168.1.216:61005] CONNECTED: datanode06/192.168.1.216:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO
ipc.NettyTransceiver: Connecting to datanode09/192.168.1.219:61005
attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO
ipc.NettyTransceiver: [id: 0x6bbccfbc] OPEN
attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO
ipc.NettyTransceiver: [id: 0x6bbccfbc, /192.168.1.211:45969 => datanode09/
192.168.1.219:61005] BOUND: /192.168.1.211:45969
attempt_201209141233_0001_000001_0: 12/09/14 12:49:04 INFO
ipc.NettyTransceiver: [id: 0x6bbccfbc, /192.168.1.211:45969 => datanode09/
192.168.1.219:61005] CONNECTED: datanode09/192.168.1.219:61005
attempt_201209141233_0001_000001_0: 12/09/14 15:08:29 INFO
clustering.KMeansBSPV02: Finished! Writing the assignments...
12/09/14 15:09:44 INFO bsp.BSPJobClient: Job failed.

All about zookeeper. I don't what happend?  Any suggestions?

btw, our cluster has 9 nodes (RAM: 4GB/node). I configured 5 tasks per
node, 512MB memory per task.

Regards,
Hu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message