giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Puneet Jain <puneetdabu...@gmail.com>
Subject Exception when running connectedcomponents on 50GB + hbase table
Date Wed, 24 Jul 2013 04:21:33 GMT
Hi .. I am getting following exceptions..

2013-07-24 00:16:42,697 INFO org.apache.giraph.comm.netty.NettyClient:
NettyClient: Using execution handler with 8 threads after requestEncoder.
2013-07-24 00:16:42,714 INFO org.apache.giraph.master.BspServiceMaster:
becomeMaster: I am now the master!
2013-07-24 00:16:42,746 INFO org.apache.giraph.bsp.BspService: process:
applicationAttemptChanged signaled
2013-07-24 00:16:42,776 WARN org.apache.giraph.bsp.BspService: process:
Unknown and unprocessed event
(path=/_hadoopBsp/job_201307231402_0017/_applicationAttemptsDir/0/_superstepDir,
type=NodeChildrenChanged, state=SyncConnected)
2013-07-24 00:16:52,501 INFO org.apache.giraph.master.BspServiceMaster:
generateVertexInputSplits: Got 16 input splits for 83 input threads
2013-07-24 00:16:52,501 WARN org.apache.giraph.master.BspServiceMaster:
createVertexInputSplits: Number of inputSplits=16 < 83=total number of
input threads, some threads will be not used
2013-07-24 00:16:52,501 INFO org.apache.giraph.master.BspServiceMaster:
createVertexInputSplits: Starting to write input split data to zookeeper
with 1 threads
2013-07-24 00:16:52,582 INFO org.apache.giraph.master.BspServiceMaster:
createVertexInputSplits: Done writing input split data to zookeeper
2013-07-24 00:16:52,743 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,760 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,767 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,767 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,768 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,771 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,772 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,772 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,773 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,772 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,777 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,777 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,780 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,780 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,780 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,781 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,781 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,781 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,781 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,782 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,786 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,786 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,786 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,787 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,787 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,787 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,788 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,788 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,789 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,790 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,790 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,791 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,792 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,792 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,792 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 00:16:52,793 INFO org.apache.giraph.comm.netty.NettyClient:
Using Netty without authentication.
2013-07-24 00:16:52,793 WARN
org.apache.giraph.comm.netty.handler.ResponseClientHandler:
exceptionCaught: Channel failed with remote address null
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)


I have 21 node hadoop/dfs cluster, 21 regionservers, 3 ZK.

My hbase input table has over 75M rows.

Thanks
Puneet

Mime
View raw message