hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Could not get distributed Hbase stable, stopping every ~24 hours.
Date Tue, 26 Aug 2014 09:26:28 GMT
Please take a look at 15.9.2.7 under
http://hbase.apache.org/book.html#trouble.rs.runtime

Cheers


On Tue, Aug 26, 2014 at 1:32 AM, Ron van der Vegt <
ron.van.der.vegt@openindex.io> wrote:

> First off all thanks for your reply. We are using three zookeeper server.
> All with version 3.4.6.
> Below the logs of one of the zookeeper server. It seems that the
> connection will at some time just drop.
>
> In the meantime I will monitor heap dumps to check if there is a memory
> leakage. But my feeling tells me the memory leakage is not the cause but
> more the effect.
> Do you see anything in all those logs which we could determine what is
> happening?
>
> Thanks!
>
> 2014-08-23 07:00:19,927 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x347fc15265a00ec, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,927 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x147fc1616d200be, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,927 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x247fc16c80500d6, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x147fc1616d200bb, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x347fc15265a00eb, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x147fc1616d200bd, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x347fc15265a00ea, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x347fc15265a00ef, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,928 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x247ffe833880000, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,929 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x247ffe833880001, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,929 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x247fc16c80500d2, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,929 [myid:3] - INFO
> [SessionTracker:ZooKeeperServer@347] - Expiring session
> 0x247fc16c80500d5, timeout of 40000ms exceeded
> 2014-08-23 07:00:19,930 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fc15265a00ec
> 2014-08-23 07:00:19,930 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x147fc1616d200be
> 2014-08-23 07:00:19,930 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x247fc16c80500d6
> 2014-08-23 07:00:19,930 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x147fc1616d200bb
> 2014-08-23 07:00:19,931 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fc15265a00eb
> 2014-08-23 07:00:19,931 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x147fc1616d200bd
> 2014-08-23 07:00:19,931 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fc15265a00ea
> 2014-08-23 07:00:19,931 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fc15265a00ef
> 2014-08-23 07:00:19,935 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x247ffe833880000
> 2014-08-23 07:00:19,936 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x247ffe833880001
> 2014-08-23 07:00:19,936 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x247fc16c80500d2
> 2014-08-23 07:00:19,937 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x247fc16c80500d5
> 2014-08-23 07:00:19,984 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.152:52656 which had sessionid 0x347fc15265a00ec
> 2014-08-23 07:00:19,985 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.152:52651 which had sessionid 0x147fc1616d200be
> 2014-08-23 07:00:19,986 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.150:44649 which had sessionid 0x247fc16c80500d6
> 2014-08-23 07:00:19,990 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.153:60445 which had sessionid 0x147fc1616d200bb
> 2014-08-23 07:00:19,993 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.154:47997 which had sessionid 0x347fc15265a00eb
> 2014-08-23 07:00:19,995 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.154:47999 which had sessionid 0x147fc1616d200bd
> 2014-08-23 07:00:19,997 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.151:34773 which had sessionid 0x347fc15265a00ea
> 2014-08-23 07:00:19,998 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.153:60444 which had sessionid 0x347fc15265a00ef
> 2014-08-23 07:00:20,000 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.102:37154 which had sessionid 0x247ffe833880000
> 2014-08-23 07:00:20,002 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.102:37155 which had sessionid 0x247ffe833880001
> 2014-08-23 07:00:20,005 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.150:48086 which had sessionid 0x247fc16c80500d2
> 2014-08-23 07:00:20,006 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@218] - Ignoring unexpected
> runtime exception
> java.nio.channels.CancelledKeyException
>     at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>     at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
>     at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
>     at java.lang.Thread.run(Thread.java:744)
> 2014-08-23 07:00:20,008 [myid:3] - INFO
> [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> client /141.105.120.151:38066 which had sessionid 0x247fc16c80500d5
> 2014-08-23 07:00:20,008 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@218] - Ignoring unexpected
> runtime exception
> java.nio.channels.CancelledKeyException
>     at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>     at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
>     at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
>     at java.lang.Thread.run(Thread.java:744)
> 2014-08-23 07:00:34,185 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /141.105.120.102:37423
> 2014-08-23 07:00:34,185 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /141.105.120.102:37423
> 2014-08-23 07:00:34,189 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130001 with negotiated timeout 40000 for client /
> 141.105.120.102:37423
> 2014-08-23 07:00:34,196 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> when processing sessionid:0x347fcb5a0130001 type:delete cxid:0x5
> zxid:0x6c0000001f txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/
> vps2008.directvps.nl,60000,1408691163492 Error:KeeperErrorCode = NoNode
> for /hbase/backup-masters/vps2008.directvps.nl,60000,1408691163492
> 2014-08-23 07:00:34,409 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> when processing sessionid:0x347fcb5a0130001 type:create cxid:0x12
> zxid:0x6c00000021 txntype:-1 reqpath:n/a Error
> Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for
> /hbase/online-snapshot/acquired
> 2014-08-23 07:00:45,466 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0001
> 2014-08-23 07:00:45,875 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58692
> 2014-08-23 07:00:45,875 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58692; will be dropped if server is in r-o mode
> 2014-08-23 07:00:45,875 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58692
> 2014-08-23 07:00:45,884 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130002 with negotiated timeout 5000 for client /
> 178.21.116.224:58692
> 2014-08-23 07:00:45,887 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130002
> 2014-08-23 07:00:45,891 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58692 which had sessionid 0x347fcb5a0130002
> 2014-08-23 07:00:46,494 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58694
> 2014-08-23 07:00:46,494 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58694; will be dropped if server is in r-o mode
> 2014-08-23 07:00:46,494 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58694
> 2014-08-23 07:00:46,498 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130003 with negotiated timeout 5000 for client /
> 178.21.116.224:58694
> 2014-08-23 07:00:46,501 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130003
> 2014-08-23 07:00:46,503 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58694 which had sessionid 0x347fcb5a0130003
> 2014-08-23 07:00:47,314 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58696
> 2014-08-23 07:00:47,314 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58696; will be dropped if server is in r-o mode
> 2014-08-23 07:00:47,315 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58696
> 2014-08-23 07:00:47,318 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130004 with negotiated timeout 5000 for client /
> 178.21.116.224:58696
> 2014-08-23 07:00:47,321 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130004
> 2014-08-23 07:00:47,324 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58696 which had sessionid 0x347fcb5a0130004
> 2014-08-23 07:00:48,334 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58698
> 2014-08-23 07:00:48,335 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58698; will be dropped if server is in r-o mode
> 2014-08-23 07:00:48,335 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58698
> 2014-08-23 07:00:48,338 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130005 with negotiated timeout 5000 for client /
> 178.21.116.224:58698
> 2014-08-23 07:00:48,342 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130005
> 2014-08-23 07:00:48,345 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58698 which had sessionid 0x347fcb5a0130005
> 2014-08-23 07:00:49,399 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x148010689410000
> 2014-08-23 07:00:50,459 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0002
> 2014-08-23 07:00:51,557 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0003
> 2014-08-23 07:00:51,563 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58706
> 2014-08-23 07:00:51,563 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58706; will be dropped if server is in r-o mode
> 2014-08-23 07:00:51,563 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58706
> 2014-08-23 07:00:51,568 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130006 with negotiated timeout 5000 for client /
> 178.21.116.224:58706
> 2014-08-23 07:00:51,571 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130006
> 2014-08-23 07:00:51,575 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58706 which had sessionid 0x347fcb5a0130006
> 2014-08-23 07:00:52,855 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0004
> 2014-08-23 07:00:52,870 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0005
> 2014-08-23 07:00:54,914 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58712
> 2014-08-23 07:00:54,914 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58712; will be dropped if server is in r-o mode
> 2014-08-23 07:00:54,915 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58712
> 2014-08-23 07:00:54,918 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130007 with negotiated timeout 5000 for client /
> 178.21.116.224:58712
> 2014-08-23 07:00:54,921 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130007
> 2014-08-23 07:00:54,925 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58712 which had sessionid 0x347fcb5a0130007
> 2014-08-23 07:00:55,357 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x148010689410001
> 2014-08-23 07:00:55,997 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x148010689410002
> 2014-08-23 07:00:56,018 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0006
> 2014-08-23 07:00:57,047 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0007
> 2014-08-23 07:00:58,583 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x148010689410003
> 2014-08-23 07:00:59,642 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0008
> 2014-08-23 07:01:00,737 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x148010689410004
> 2014-08-23 07:01:01,904 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff0009
> 2014-08-23 07:01:08,186 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /178.21.116.224:58730
> 2014-08-23 07:01:08,187 [myid:3] - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> client /178.21.116.224:58730; will be dropped if server is in r-o mode
> 2014-08-23 07:01:08,187 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /178.21.116.224:58730
> 2014-08-23 07:01:08,191 [myid:3] - INFO
> [CommitProcessor:3:ZooKeeperServer@617] - Established session
> 0x347fcb5a0130008 with negotiated timeout 5000 for client /
> 178.21.116.224:58730
> 2014-08-23 07:01:08,194 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x347fcb5a0130008
> 2014-08-23 07:01:08,197 [myid:3] - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /178.21.116.224:58730 which had sessionid 0x347fcb5a0130008
> 2014-08-23 07:01:09,735 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff000a
> 2014-08-23 07:01:11,799 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> sessionid: 0x24801068aff000b
>
>
>
> -----Oorspronkelijk bericht-----
> > Afzender:Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com> >
> > Verstuurd: Maandag 25 Augustus 2014 16:45
> > Aan: user@hbase.apache.org <mailto:user@hbase.apache.org>
> > Onderwerp: Re: Could not get distributed Hbase stable, stopping every
> ˜24 hours.
> >
> > Have you checked zookeeper logs ?
> > What zookeeper release are you using ?
> >
> > bq. Could there be a memory leakage?
> >
> > You can use jmap to capture heap memory details:
> >
> > http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html <
> http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html>
> >
> > Cheers
> >
> >
> > On Mon, Aug 25, 2014 at 4:51 AM, Ron van der Vegt <
> > ron.van.der.vegt@openindex.io <mailto:ron.van.der.vegt@openindex.io> >
> wrote:
> >
> > > Hi all!
> > >
> > > I have setup one master and 5 regionservers to collect log data. But
> every
> > > ˜24 hours, at random times, the regionservers generating a fatal error
> and
> > > all stopping one by one. Eventually the master will stop. I also see
> some
> > > weird characters before the server names in the logs. Seems like some
> > > encoding issue.
> > >
> > > I have read in the documentation, that if the garbage collection is
> taking
> > > to long, you will also get the session expired message. But I have
> logged
> > > the GC on the master, and it seems oke. Could someone help me figure
> out
> > > why this is happening?
> > >
> > > Furthermore, I am currently monitoring the memory usage of the master
> with
> > > JMX. I notice that the heap size is slowly growing. Could there be a
> memory
> > > leakage?
> > > xmx is set to 1gb.
> > >
> > > Setup:
> > > hbase 0.94.20
> > > hadoop 1.2.1
> > > debian wheezy
> > >
> > > Thanks in advice,
> > >
> > > Ron
> > >
> > > Logs of master:
> > > ===============
> > >
> > > 2014-08-23 07:00:20,104 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > > ZooKeeper exception:
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase/unassigned/70236052
> > > 2014-08-23 07:00:20,406 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2060.directvps.nl,60020,1408691165501 reported a
> fatal
> > > error:
> > > ABORTING region server vps2060.directvps.nl,60020,1408691165501:
> > >
> regionserver:60020-0x347fc15265a00eb-0x347fc15265a00eb-0x347fc15265a00eb
> > >
> regionserver:60020-0x347fc15265a00eb-0x347fc15265a00eb-0x347fc15265a00eb
> > > received expired from ZooKeeper, aborting
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:20,911 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2057.directvps.nl,60020,1408691165499 reported a
> fatal
> > > error:
> > > ABORTING region server vps2057.directvps.nl,60020,1408691165499:
> > >
> regionserver:60020-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea
> > >
> regionserver:60020-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea
> > > received expired from ZooKeeper, aborting
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:21,001 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2059.directvps.nl,60020,1408691165851 reported a
> fatal
> > > error:
> > > ABORTING region server vps2059.directvps.nl,60020,1408691165851:
> > >
> regionserver:60020-0x147fc1616d200bb-0x147fc1616d200bb-0x147fc1616d200bb
> > >
> regionserver:60020-0x147fc1616d200bb-0x147fc1616d200bb-0x147fc1616d200bb
> > > received expired from ZooKeeper, aborting
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:21,056 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2058.directvps.nl,60020,1408691165675 reported a
> fatal
> > > error:
> > > ABORTING region server vps2058.directvps.nl,60020,1408691165675:
> > >
> regionserver:60020-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec
> > >
> regionserver:60020-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec
> > > received expired from ZooKeeper, aborting
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:22,140 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > > ZooKeeper exception:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > 2014-08-23 07:00:26,141 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > > ZooKeeper exception:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > 2014-08-23 07:00:34,114 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2056.directvps.nl,60020,1408691165439 reported a
> fatal
> > > error:
> > > ABORTING region server vps2056.directvps.nl,60020,1408691165439:
> > > Unexpected exception handling nodeDeleted event
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/master
> > >     at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > >     at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >     at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:182)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:34,118 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > Region server vps2056.directvps.nl,60020,1408691165439 reported a
> fatal
> > > error:
> > > ABORTING region server vps2056.directvps.nl,60020,1408691165439:
> > >
> regionserver:60020-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2
> > >
> regionserver:60020-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2
> > > received expired from ZooKeeper, aborting
> > > Cause:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > >
> > > 2014-08-23 07:00:34,141 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > > ZooKeeper exception:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > 2014-08-23 07:00:34,142 ERROR
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
> getData
> > > failed after 3 retries
> > > 2014-08-23 07:00:34,152 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > >
> master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001
> > > Unable to get data of znode /hbase/unassigned/70236052
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > >     at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > >     at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > >     at
> > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > >     at
> > >
> org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > 2014-08-23 07:00:34,152 ERROR
> > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
> > >
> master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001
> > > Received unexpected KeeperException, re-throwing exception
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > >     at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > >     at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > >     at
> > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > >     at
> > >
> org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > 2014-08-23 07:00:34,163 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Master server abort: loaded coprocessors are: []
> > > 2014-08-23 07:00:34,215 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > > /hbase/backup-masters/vps2008.directvps.nl,60000,1408691163492 already
> > > deleted, and this is not a retry
> > > 2014-08-23 07:05:34,165 WARN
> > > org.apache.hadoop.hbase.master.SplitLogManager: Interrupted while
> waiting
> > > for log splits to be completed
> > > 2014-08-23 07:05:34,179 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Unexpected ZK exception reading unassigned node for region=70236052
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > >     at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > >     at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > >     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > >     at
> > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > >     at
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > >     at
> > >
> org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > 2014-08-23 07:05:34,179 WARN
> > > org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
> logs
> > > in [hdfs://
> > >
> namenode.openindex.io:8020/hbase/.logs/vps2058.directvps.nl,60020,1408691165675-splitting
> ]
> > > installed = 2 but only 0 done
> > > 2014-08-23 07:05:34,184 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping
> > > server on 60000
> > > 2014-08-23 07:05:34,185 WARN
> > > org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog
> table
> > > java.io.IOException: Giving up after tries=1
> > >     at
> > >
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:210)
> > >     at
> > >
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:188)
> > >     at
> > >
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
> > >     at
> > >
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:67)
> > >     at
> > >
> org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:126)
> > >     at
> > >
> org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:137)
> > >     at
> > >
> org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
> > >     at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
> > >     at java.lang.Thread.run(Thread.java:745)
> > > Caused by: java.lang.InterruptedException: sleep interrupted
> > >     at java.lang.Thread.sleep(Native Method)
> > >     at
> > >
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:207)
> > >     ... 8 more
> > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 5 on 60000: exiting
> > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 6 on 60000: exiting
> > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> IPC
> > > Server handler 0 on 60000: exiting
> > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 2 on 60000: exiting
> > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 3 on 60000: exiting
> > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 4 on 60000: exiting
> > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 7 on 60000: exiting
> > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 8 on 60000: exiting
> > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 9 on 60000: exiting
> > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> IPC
> > > Server handler 2 on 60000: exiting
> > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping
> > > IPC Server listener on 60000
> > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping
> > > IPC Server Responder
> > > 2014-08-23 07:05:34,214 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping
> > > IPC Server Responder
> > > 2014-08-23 07:05:34,212 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> IPC
> > > Server handler 1 on 60000: exiting
> > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 0 on 60000: exiting
> > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > > handler 1 on 60000: exiting
> > > 2014-08-23 07:05:34,256 INFO org.mortbay.log: Stopped
> > > SelectChannelConnector@0.0.0.0:60010
> > > 2014-08-23 07:05:34,259 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Master server abort: loaded coprocessors are: []
> > > 2014-08-23 07:05:34,260 FATAL org.apache.hadoop.hbase.master.HMaster:
> > >
> master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001-0x347fcb5a0130001
> > >
> master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001-0x347fcb5a0130001
> > > received expired from ZooKeeper, aborting
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > >     at
> > >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > >     at
> > >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > >     at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > 2014-08-23 07:05:34,414 ERROR
> > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start
> master
> > > java.lang.RuntimeException: HMaster Aborted
> > >     at
> > >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> > >     at
> > >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> > >     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > >     at
> > >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> > >     at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2129)
> > >
> > > GC log:
> > > =======
> > >
> > > 0.185: Application time: 0.1304320 seconds
> > > 0.185: [GC0.185: [ParNew: 4288K->511K(4800K), 0.0120520 secs]
> > > 4288K->1008K(15424K), 0.0121600 secs] [Times: user=0.01 sys=0.01,
> real=0.01
> > > secs]
> > > 0.197: Total time for which application threads were stopped: 0.0126240
> > > seconds
> > > Heap
> > >  par new generation   total 4800K, used 3580K [0x00000000b7200000,
> > > 0x00000000b7730000, 0x00000000c1860000)
> > >   eden space 4288K,  71% used [0x00000000b7200000, 0x00000000b74ff328,
> > > 0x00000000b7630000)
> > >   from space 512K,  99% used [0x00000000b76b0000, 0x00000000b772fff8,
> > > 0x00000000b7730000)
> > >   to   space 512K,   0% used [0x00000000b7630000, 0x00000000b7630000,
> > > 0x00000000b76b0000)
> > >  concurrent mark-sweep generation total 10624K, used 496K
> > > [0x00000000c1860000, 0x00000000c22c0000, 0x00000000f5a00000)
> > >  concurrent-mark-sweep perm gen total 21248K, used 6688K
> > > [0x00000000f5a00000, 0x00000000f6ec0000, 0x0000000100000000)
> > > 0.370: Application time: 0.1728650 seconds
> > >
> > >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message