hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: What is the best configuration for Cluster hama distributed mode
Date Thu, 27 Dec 2012 13:34:59 GMT
You should increase max heap size of child JVMs, instead of HAMA_HEAPSIZE.

Add below property to hama-site.xml.

  <property>
    <name>bsp.child.java.opts</name>
    <value>-Xmx2048m</value>
    <description>Java opts for the groom server child processes.
    The following symbol, if present, will be interpolated: @taskid@
is replaced
    by current TaskID. Any other occurrences of '@' will go unchanged.
    For example, to enable verbose gc logging to a file named for the taskid in
    /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
          -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
    The configuration variable bsp.child.ulimit can be used to control the
    maximum virtual memory of the child processes.
    </description>
  </property>

On Thu, Dec 27, 2012 at 5:11 AM, Francisco Sanches
<sanchesbanhos@gmail.com> wrote:
> Thanks for the replies:
>
> Excuse me by the size of the log. But after I changed the parameter of
> hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving the
> memory error and displays the error below. Remembering that I am running
> for a graph with just over 65 000 vertices. What is it like to understand
> is that if I increase the number of nodes in the cluster can run my
> application with this graph of 65 000 vertices?
>
> LOG:
>
> hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537
> /hdfs/disco/grafogrande /hdfs/disco/grafoout 36
> 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process : 1
> 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36
> 12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job: job_201212261658_0001
> 12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0
> 12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
> built on 02/23/2011 22:27 GMT
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:host.name=mestre
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.name=Linux
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.arch=i386
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:user.name=hduser
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Initiating client connection,
> connectString=node02:21810,node01:21810,mestre:21810 sessionTimeout=1200000
> watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ClientCnxn: Opening socket connection to server mestre/
> 10.16.1.200:21810
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ClientCnxn: Socket connection established to mestre/
> 10.16.1.200:21810, initiating session
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO
> zookeeper.ClientCnxn: Session establishment complete on server mestre/
> 10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout =
> 1200000
> attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple
> SLF4J bindings.
> attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> [jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> [file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class]
> attempt_201212261658_0001_000000_0: SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> 10.16.1.203:61003] BOUND: /10.16.1.200:47685
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> 10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> 10.16.1.205:61001] BOUND: /10.16.1.200:52329
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> 10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> 10.16.1.204:61002] BOUND: /10.16.1.200:59179
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> 10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> 10.16.1.206:61001] BOUND: /10.16.1.200:45011
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> 10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> 10.16.1.204:61003] BOUND: /10.16.1.200:54578
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> 10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> 10.16.1.205:61002] BOUND: /10.16.1.200:52964
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> 10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> 10.16.1.203:61001] BOUND: /10.16.1.200:36765
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> 10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> 10.16.1.201:61003] BOUND: /10.16.1.200:45734
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> 10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> 10.16.1.202:61002] BOUND: /10.16.1.200:60724
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> 10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> 10.16.1.203:61002] BOUND: /10.16.1.200:45839
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> 10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> 10.16.1.202:61003] BOUND: /10.16.1.200:44757
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> 10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> 10.16.1.204:61001] BOUND: /10.16.1.200:40846
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> 10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.210:36396
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41941
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44493
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48243
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41945
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> 10.16.1.201:61001] BOUND: /10.16.1.200:35763
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> 10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41949
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> 10.16.1.200:61002] BOUND: /10.16.1.200:41949
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> 10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43193
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> 10.16.1.201:61002] BOUND: /10.16.1.200:44908
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> 10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57476
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57478
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.210:36402
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37455
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57480
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56188
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44506
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56189
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48251
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56191
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50918
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50917
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43200
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43201
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50924
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44099
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44511
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44100
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44101
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37460
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> 10.16.1.202:61001] BOUND: /10.16.1.200:54333
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> 10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46568
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46569
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> 10.16.1.200:61003] BOUND: /10.16.1.200:56327
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> 10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46573
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> 10.16.1.200:61001] BOUND: /10.16.1.200:51740
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> 10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35237
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35239
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48260
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> 10.16.1.211:61003] BOUND: /10.16.1.200:57399
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> 10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37463
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> 10.16.1.211:61001] BOUND: /10.16.1.200:41092
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> 10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35245
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> 10.16.1.210:61002] BOUND: /10.16.1.200:48018
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> 10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> 10.16.1.209:61003] BOUND: /10.16.1.200:34916
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> 10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> 10.16.1.211:61002] BOUND: /10.16.1.200:58261
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> 10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> 10.16.1.210:61003] BOUND: /10.16.1.200:39073
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> 10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> 10.16.1.208:61002] BOUND: /10.16.1.200:39758
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> 10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> 10.16.1.209:61001] BOUND: /10.16.1.200:56029
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> 10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> 10.16.1.207:61003] BOUND: /10.16.1.200:35151
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> 10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> 10.16.1.208:61003] BOUND: /10.16.1.200:35301
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> 10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> 10.16.1.210:61001] BOUND: /10.16.1.200:34819
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> 10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> 10.16.1.209:61002] BOUND: /10.16.1.200:34684
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> 10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> 10.16.1.206:61002] BOUND: /10.16.1.200:35962
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> 10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> 10.16.1.207:61001] BOUND: /10.16.1.200:41009
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> 10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> 10.16.1.205:61003] BOUND: /10.16.1.200:53654
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> 10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> 10.16.1.208:61001] BOUND: /10.16.1.200:55138
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> 10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> 10.16.1.206:61003] BOUND: /10.16.1.200:40540
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> 10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> 10.16.1.207:61002] BOUND: /10.16.1.200:42816
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> 10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO
> graph.GraphJobRunner: Loading finished at 1 steps.
> 12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed.
>
>
>
> 2012/12/26 Edward J. Yoon <edwardyoon@apache.org>
>
>> BTW, This log looks like you ran out of memory while job was running
>> as a local mode.
>>
>> You can check the cluster status via web UI.
>>
>> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces
>>
>> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
>> <sanchesbanhos@gmail.com> wrote:
>> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
>> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Francisco Sanches



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message