hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Luo <j...@merkleinc.com>
Subject what causes hbase client to open large number of connectoins?
Date Fri, 09 Sep 2016 23:52:06 GMT
I have observed a very weird behavior that crashes zookeepers.

I have one job that uses 100 reducers to perform puts. The same code works fine with all other
users/data, but just on a particular user with its particular data, it crashes zookeepers
and bring down the entire cluster (wooh).

The log shows 140 “Opening socket connection” operation on just one failed attempt as
shown below. I can understand that 140 * 100 = 14K connections could crash zookeepers, but
my question is what cause it to open so many connections? The other successful run with the
same logic opens one or two connections.


My code just do “connection.getTable(TableName)” five times to create five tables for
each attempt.



Here is a part of log, showing many “Opening socket connections”.

==========================================================

2016-09-07 15:43:56,745 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Opening socket connection to server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not
attempt to authenticate using SASL (unknown error)
2016-09-07 15:43:56,746 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Socket connection established to hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating
session
2016-09-07 15:43:56,747 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Session establishment complete on server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid
= 0x257027220206f90, negotiated timeout = 120000
2016-09-07 15:44:22,530 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x2d90f90b connecting to ZooKeeper ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
2016-09-07 15:44:22,530 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection,
connectString=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
sessionTimeout=120000 watcher=hconnection-0x2d90f90b0x0, quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
baseZNode=/hbase-secure
2016-09-07 15:44:22,530 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Opening socket connection to server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not
attempt to authenticate using SASL (unknown error)
2016-09-07 15:44:22,531 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Socket connection established to hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating
session
2016-09-07 15:44:22,532 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Session establishment complete on server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid
= 0x257027220206fe3, negotiated timeout = 120000
2016-09-07 15:45:09,081 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x43e65f2a connecting to ZooKeeper ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
2016-09-07 15:45:09,081 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection,
connectString=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
sessionTimeout=120000 watcher=hconnection-0x43e65f2a0x0, quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
baseZNode=/hbase-secure
2016-09-07 15:45:09,082 INFO [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] org.apache.zookeeper.ClientCnxn:
Opening socket connection to server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not
attempt to authenticate using SASL (unknown error)



Access the Q2 2016 Digital Marketing Report for a fresh set of trends and benchmarks in digital
marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c>

Download our latest report titled “The Case for Change: Exploring the Myths of Customer-Centric
Transformation” <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c>

This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message