hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiang Hua <bea...@gmail.com>
Subject Re: 答复: hmaster down cause by zookeeper?
Date Sat, 13 Oct 2012 01:55:51 GMT
Hi,
    below is hbase log, and connection problem with ZK. please help me find
if there is some problem?


2012-10-12 00:00:19,582 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:00:19,583 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, initiating session
2012-10-12 00:00:19,584 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, sessionid = 0x139c539bc090878, negotiated timeout = 40000
2012-10-12 00:00:19,588 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row= for max=2147483647 rows
2012-10-12 00:00:19,589 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1c0ec83c;
hsa=bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7:60020
2012-10-12 00:00:19,591 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9:60020
2012-10-12 00:00:19,594 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x139c539bc090878 has been closed.
2012-10-12 00:00:19,594 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x139c539bc090878
2012-10-12 00:00:19,595 INFO org.apache.zookeeper.ZooKeeper: Session:
0x139c539bc090878 closed
2012-10-12 00:00:19,595 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:01:37,861 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 4 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2012-10-12 00:01:37,888 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information:
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9,60020,1347901025670=1,
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3,60020,1347901025664=1,
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10,60020,1347901025664=0,
bj-ecsxhm4f3I-r3-7-r810-1-hbase-stor-8,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1,60020,1347901025671=1,
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6,60020,1347901025662=0,
bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673=1
2012-10-12 00:01:37,889 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=8 regions=6 average=0.75 mostloaded=1
leastloaded=0
2012-10-12 00:02:37,993 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
connectString=bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1:2181,bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3:2181,bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2:2181
sessionTimeout=180000 watcher=hconnection
2012-10-12 00:02:37,994 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:02:37,995 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:02:37,996 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, sessionid = 0x339c539ba641db9, negotiated timeout = 40000
2012-10-12 00:02:38,000 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row= for max=2147483647 rows
2012-10-12 00:02:38,000 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@41d55252;
hsa=bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7:60020
2012-10-12 00:02:38,002 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9:60020
2012-10-12 00:02:38,004 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x339c539ba641db9 has been closed.
2012-10-12 00:02:38,004 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x339c539ba641db9
2012-10-12 00:02:38,005 INFO org.apache.zookeeper.ZooKeeper: Session:
0x339c539ba641db9 closed
2012-10-12 00:02:38,005 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:04:34,331 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
connectString=bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1:2181,bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3:2181,bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2:2181
sessionTimeout=180000 watcher=hconnection
2012-10-12 00:04:34,332 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:04:34,332 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181, initiating session
2012-10-12 00:04:34,337 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181, sessionid = 0x239c539ba632f30, negotiated timeout = 40000
2012-10-12 00:04:34,342 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row= for max=2147483647 rows
2012-10-12 00:04:34,342 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6a9d547c;
hsa=bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7:60020
2012-10-12 00:04:34,343 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9:60020
2012-10-12 00:04:34,346 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to hconnection-0x239c539ba632f30 has been closed.
2012-10-12 00:04:34,346 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x239c539ba632f30
2012-10-12 00:04:34,347 INFO org.apache.zookeeper.ZooKeeper: Session:
0x239c539ba632f30 closed
2012-10-12 00:04:34,347 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:05:37,043 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null has been closed.
2012-10-12 00:05:37,043 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null was closed by the finalize method.
2012-10-12 00:05:37,043 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null has been closed.
2012-10-12 00:05:37,043 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null was closed by the finalize method.
2012-10-12 00:05:37,044 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null has been closed.
2012-10-12 00:05:37,044 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
The connection to null was closed by the finalize method.
2012-10-12 00:06:37,898 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 4 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2012-10-12 00:06:37,927 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information:
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9,60020,1347901025670=1,
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3,60020,1347901025664=1,
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10,60020,1347901025664=0,
bj-ecsxhm4f3I-r3-7-r810-1-hbase-stor-8,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1,60020,1347901025671=1,
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6,60020,1347901025662=0,
bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673=1
2012-10-12 00:06:37,927 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=8 regions=6 average=0.75 mostloaded=1
leastloaded=0
2012-10-12 00:07:28,856 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1347901026386
2012-10-12 00:07:28,859 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1348729270029
2012-10-12 00:07:28,863 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349413471232
2012-10-12 00:07:28,865 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349417072305
2012-10-12 00:07:28,868 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349420673379
2012-10-12 00:07:28,871 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349424274450
2012-10-12 00:07:28,874 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349427875522
2012-10-12 00:07:28,877 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349431476597
2012-10-12 00:07:28,880 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349435077672
2012-10-12 00:07:28,883 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349438678745
2012-10-12 00:07:28,886 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3%3A60020.1349442279817
2012-10-12 00:07:28,889 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1347901026386
2012-10-12 00:07:28,891 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1348729266284
2012-10-12 00:07:28,894 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349413464398
2012-10-12 00:07:28,897 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349417065468
2012-10-12 00:07:28,900 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349420666539
2012-10-12 00:07:28,903 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349424267608
2012-10-12 00:07:28,906 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349427868675
2012-10-12 00:07:28,909 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349431469744
2012-10-12 00:07:28,912 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349435070812
2012-10-12 00:07:28,915 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349438671880
2012-10-12 00:07:28,918 DEBUG
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Didn't
find this log in ZK, deleting:
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10%3A60020.1349442272948
2012-10-12 00:11:37,937 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 4 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2012-10-12 00:11:37,966 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information:
bj-ecsxhm4f3I-r3-8-r810-4-hbase-stor-9,60020,1347901025670=1,
bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3,60020,1347901025664=1,
bj-ecsxhm4f3I-r3-8-r810-3-hbase-stor-10,60020,1347901025664=0,
bj-ecsxhm4f3I-r3-7-r810-1-hbase-stor-8,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1,60020,1347901025671=1,
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6,60020,1347901025662=0,
bj-ecsxhm4f3I-r3-7-r810-2-hbase-stor-7,60020,1347901025661=1,
bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673=1
2012-10-12 00:11:37,966 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=8 regions=6 average=0.75 mostloaded=1
leastloaded=0
2012-10-12 00:13:39,946 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 26666ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:13:40,978 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:13:42,242 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 26666ms for sessionid
0x139c539bc090003, closing socket connection and attempting reconnect
2012-10-12 00:13:42,974 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:13:46,954 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 26666ms for sessionid
0x139c539bc090002, closing socket connection and attempting reconnect
2012-10-12 00:13:47,516 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:13:54,311 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 14262ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:13:55,257 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:13:56,306 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13962ms for sessionid
0x139c539bc090003, closing socket connection and attempting reconnect
2012-10-12 00:13:56,493 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:14:00,849 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13793ms for sessionid
0x139c539bc090002, closing socket connection and attempting reconnect
2012-10-12 00:14:01,946 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:14:08,590 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 14176ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:09,826 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13418ms for sessionid
0x139c539bc090003, closing socket connection and attempting reconnect
2012-10-12 00:14:10,404 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:14:11,840 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:14:15,279 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 14328ms for sessionid
0x139c539bc090002, closing socket connection and attempting reconnect
2012-10-12 00:14:16,445 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, initiating session
2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout = 40000
2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15046ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15245ms for sessionid
0x139c539bc090003, closing socket connection and attempting reconnect
2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x139c539bc090003 has expired,
closing socket connection
2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session
0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
    at sun.nio.ch.IOUtil.read(IOUtil.java:186)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 14007ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/
10.20.16.32:2181
2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 26666ms for sessionid
0x139c539bc090002, closing socket connection and attempting reconnect
2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:14:46,572 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, initiating session
2012-10-12 00:14:46,726 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181, sessionid = 0x139c539bc090002, negotiated timeout = 40000
2012-10-12 00:14:54,925 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13464ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:14:56,524 ERROR org.apache.hadoop.hbase.master.HMaster:
Region server
serverName=bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673,
load=(requests=75, regions=1, usedHeap=162, maxHeap=9725) reported a fatal
error:
ABORTING region server
serverName=bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2,60020,1347901025673,
load=(requests=75, regions=1, usedHeap=162, maxHeap=9725):
regionserver:60020-0x339c539ba640003 regionserver:60020-0x339c539ba640003
received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:353)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
    at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507)

2012-10-12 00:14:56,813 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/
10.20.16.33:2181
2012-10-12 00:15:10,147 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 15119ms for sessionid
0x239c539ba630001, closing socket connection and attempting reconnect
2012-10-12 00:15:10,625 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181
2012-10-12 00:15:10,625 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/
10.20.16.34:2181, initiating session
2012-10-12 00:15:10,750 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x239c539ba630001 has expired,
closing socket connection
2012-10-12 00:15:10,750 FATAL org.apache.hadoop.hbase.master.HMaster:
master:60000-0x239c539ba630001 master:60000-0x239c539ba630001 received
expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:353)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
    at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507)
2012-10-12 00:15:10,751 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2012-10-12 00:15:10,751 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2012-10-12 00:15:11,392 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2012-10-12 00:15:11,392 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2012-10-12 00:15:11,392 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6:60000-CatalogJanitor exiting
2012-10-12 00:15:11,392 INFO org.apache.hadoop.hbase.master.HMaster$2:
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6:60000-BalancerChore exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 25 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 29 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 15 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 37 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 40 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 41 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 46 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 47 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 50 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 51 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60000: exiting
2012-10-12 00:15:11,393 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 59 on 60000: exiting
2012-10-12 00:15:11,395 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6:60000.timeoutMonitor exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 53 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 54 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 58 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 57 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 56 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 55 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 52 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 49 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 48 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 44 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 43 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 45 on 60000: exiting
2012-10-12 00:15:11,395 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 42 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 39 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 38 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 35 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 36 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 34 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 32 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 33 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.mortbay.log: Stopped
SelectChannelConnector@0.0.0.0:60010
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 30 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 31 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 28 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 27 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 26 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60000: exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.hbase.master.LogCleaner:
master-bj-ecsxhm4f3I-r3-7-r810-3-hbase-stor-6:60000.oldLogCleaner exiting
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60000: exiting
2012-10-12 00:15:11,398 INFO
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Stopping
replicationLogCleaner-0x139c539bc090003
2012-10-12 00:15:11,394 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60000: exiting
2012-10-12 00:15:11,502 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x239c539ba630001 Unable to get data of znode /hbase/master
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/master

On Fri, Oct 12, 2012 at 12:16 PM, 谢良 <xieliang@xiaomi.com> wrote:

> Hi Xiang,
>
> It's not the root cause,  if you skim through sendBuffer impl in
> NIOServerCnxn.java, you'll find there's a catch statement finally to log
> all exception, no throw again.
>
> IMHO, the hbase master log file is the right place you need to dive:)
> ________________________________________
> 发件人: Xiang Hua [beatls@gmail.com]
> 发送时间: 2012年10月12日 3:14
> 收件人: user@hbase.apache.org
> 主题: hmaster down cause by zookeeper?
>
> Hi,
>     we using CDH3, using zookeeper,hdfs and hbase.
>     now 2 hmaster is down, we find some error message from zookeeper, see
> below:
>
> 2012-10-12 02:02:53,928 - ERROR [CommitProcessor:1:NIOServerCnxn@445
> ] - Unexpected Exception:
> java.nio.channels.CancelledKeyException
>         at
> sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>         at
> sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
>         at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367)
>         at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2012-10-12 02:02:53,928 - ERROR [CommitProcessor:1:NIOServerCnxn@445
> ] - Unexpected Exception:
> java.nio.channels.CancelledKeyException
>         at
> sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>         at
> sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
>         at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367)
>         at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2:31:34
>
>       any reconmends?
>
>     Thanks!
>
> beatls
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message