Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: Pass (protection.outlook.com: domain of merkleinc.com designates
 65.210.57.1 as permitted sender) receiver=protection.outlook.com;
 client-ip=65.210.57.1; helo=HQITEXCH05.pclc0.merkle.local;
From: Frank Luo <jluo@merkleinc.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: what causes hbase client to open large number of connectoins?
Thread-Topic: what causes hbase client to open large number of connectoins?
Thread-Index: AdIK8pj5y/Kv0R/KREybE4O3QUBevA==
Date: Fri, 9 Sep 2016 23:52:06 +0000
Message-ID: <1EFFC0EDF85B044A99A02C925FDA37552022574E@HQITEXCH07.pclc0.merkle.local>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_1EFFC0EDF85B044A99A02C925FDA37552022574EHQITEXCH07pclc0_"
MIME-Version: 1.0
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2016 23:52:07.7884
 (UTC)
X-MS-Exchange-CrossTenant-Id: 1396be20-c06a-4b1f-811d-98bb02e25b5a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=1396be20-c06a-4b1f-811d-98bb02e25b5a;Ip=[65.210.57.1];Helo=[HQITEXCH05.pclc0.merkle.local]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR06MB2161
archived-at: Fri, 09 Sep 2016 23:52:22 -0000

--_000_1EFFC0EDF85B044A99A02C925FDA37552022574EHQITEXCH07pclc0_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

I have observed a very weird behavior that crashes zookeepers.

I have one job that uses 100 reducers to perform puts. The same code works =
fine with all other users/data, but just on a particular user with its part=
icular data, it crashes zookeepers and bring down the entire cluster (wooh)=
.

The log shows 140 =E2=80=9COpening socket connection=E2=80=9D operation on =
just one failed attempt as shown below. I can understand that 140 * 100 =3D=
 14K connections could crash zookeepers, but my question is what cause it t=
o open so many connections? The other successful run with the same logic op=
ens one or two connections.


My code just do =E2=80=9Cconnection.getTable(TableName)=E2=80=9D five times=
 to create five tables for each attempt.


Here is a part of log, showing many =E2=80=9COpening socket connections=E2=
=80=9D.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D

2016-09-07 15:43:56,745 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to serve=
r hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not attempt to authe=
nticate using SASL (unknown error)
2016-09-07 15:43:56,746 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established to h=
qhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating session
2016-09-07 15:43:56,747 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on =
server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid =3D 0x2570=
27220206f90, negotiated timeout =3D 120000
2016-09-07 15:44:22,530 INFO [main] org.apache.hadoop.hbase.zookeeper.Recov=
erableZooKeeper: Process identifier=3Dhconnection-0x2d90f90b connecting to =
ZooKeeper ensemble=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.me=
rkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
2016-09-07 15:44:22,530 INFO [main] org.apache.zookeeper.ZooKeeper: Initiat=
ing client connection, connectString=3Dhqhd02nm01.pclc0.merkle.local:2181,h=
qhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 sessio=
nTimeout=3D120000 watcher=3Dhconnection-0x2d90f90b0x0, quorum=3Dhqhd02nm01.=
pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0=
.merkle.local:2181, baseZNode=3D/hbase-secure
2016-09-07 15:44:22,530 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to serve=
r hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not attempt to authe=
nticate using SASL (unknown error)
2016-09-07 15:44:22,531 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established to h=
qhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating session
2016-09-07 15:44:22,532 INFO [main-SendThread(hqhd02nm01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on =
server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid =3D 0x2570=
27220206fe3, negotiated timeout =3D 120000
2016-09-07 15:45:09,081 INFO [main] org.apache.hadoop.hbase.zookeeper.Recov=
erableZooKeeper: Process identifier=3Dhconnection-0x43e65f2a connecting to =
ZooKeeper ensemble=3Dhqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.pclc0.me=
rkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
2016-09-07 15:45:09,081 INFO [main] org.apache.zookeeper.ZooKeeper: Initiat=
ing client connection, connectString=3Dhqhd02nm01.pclc0.merkle.local:2181,h=
qhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181 sessio=
nTimeout=3D120000 watcher=3Dhconnection-0x43e65f2a0x0, quorum=3Dhqhd02nm01.=
pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0=
.merkle.local:2181, baseZNode=3D/hbase-secure
2016-09-07 15:45:09,082 INFO [main-SendThread(hqhd02ed01.pclc0.merkle.local=
:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to serve=
r hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to authe=
nticate using SASL (unknown error)


Access the Q2 2016 Digital Marketing Report for a fresh set of trends and b=
enchmarks in digital marketing<http://www2.merkleinc.com/l/47252/2016-07-26=
/47gt7c>

Download our latest report titled =E2=80=9CThe Case for Change: Exploring t=
he Myths of Customer-Centric Transformation=E2=80=9D <http://www2.merkleinc=
.com/l/47252/2016-08-04/4b9p7c>

This email and any attachments transmitted with it are intended for use by =
the intended recipient(s) only. If you have received this email in error, p=
lease notify the sender immediately and then delete it. If you are not the =
intended recipient, you must not keep, use, disclose, copy or distribute th=
is email without the author=E2=80=99s prior permission. We take precautions=
 to minimize the risk of transmitting software viruses, but we advise you t=
o perform your own virus checks on any attachment to this message. We canno=
t accept liability for any loss or damage caused by software viruses. The i=
nformation contained in this communication may be confidential and may be s=
ubject to the attorney-client privilege.

--_000_1EFFC0EDF85B044A99A02C925FDA37552022574EHQITEXCH07pclc0_--