hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Luo <j...@merkleinc.com>
Subject RE: what causes hbase client to open large number of connectoins?
Date Sat, 10 Sep 2016 05:05:19 GMT
I have counted the number of "Opening socket connection" and there are 140 of them in one reducer
log. So when I have 100 reducers running, which I do, then there are 140 * 100 = 14K  connections
alive.

Anyway, I did discover that due to that specific data, I do have a situation that I called
"ConnectionFactory.createConnection" many many times, which I shouldn't. So the mystery is
resolved.

I am just amazed that I did manage to crash ZK. :(

-----Original Message-----
From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
Sent: Friday, September 09, 2016 7:57 PM
To: user@hbase.apache.org
Subject: Re: what causes hbase client to open large number of connectoins?

They are 30-40 sec apart. Can you tell us how did you come up with 14K connections?

-Vlad

On Fri, Sep 9, 2016 at 4:52 PM, Frank Luo <jluo@merkleinc.com> wrote:

> I have observed a very weird behavior that crashes zookeepers.
>
> I have one job that uses 100 reducers to perform puts. The same code
> works fine with all other users/data, but just on a particular user
> with its particular data, it crashes zookeepers and bring down the
> entire cluster (wooh).
>
> The log shows 140 “Opening socket connection” operation on just one
> failed attempt as shown below. I can understand that 140 * 100 = 14K
> connections could crash zookeepers, but my question is what cause it
> to open so many connections? The other successful run with the same
> logic opens one or two connections.
>
>
> My code just do “connection.getTable(TableName)” five times to create
> five tables for each attempt.
>
>
>
> Here is a part of log, showing many “Opening socket connections”.
>
> ==========================================================
>
> 2016-09-07 15:43:56,745 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Opening socket connection to server
> hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-09-07 15:43:56,746 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Socket connection established to
> hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating session
> 2016-09-07 15:43:56,747 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid =
> 0x257027220206f90, negotiated timeout = 120000
> 2016-09-07 15:44:22,530 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x2d90f90b connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 2016-09-07 15:44:22,530 INFO [main] org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=hqhd02nm01.
> pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:
> 2181,hqhd02ed01.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=hconnection-0x2d90f90b0x0, quorum=hqhd02nm01.pclc0.
> merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,
> hqhd02ed01.pclc0.merkle.local:2181, baseZNode=/hbase-secure
> 2016-09-07 15:44:22,530 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Opening socket connection to server
> hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-09-07 15:44:22,531 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Socket connection established to
> hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, initiating session
> 2016-09-07 15:44:22,532 INFO
> [main-SendThread(hqhd02nm01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server hqhd02nm01.pclc0.merkle.local/10.129.8.13:2181, sessionid =
> 0x257027220206fe3, negotiated timeout = 120000
> 2016-09-07 15:45:09,081 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x43e65f2a connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 2016-09-07 15:45:09,081 INFO [main] org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=hqhd02nm01.
> pclc0.merkle.local:2181,hqhd02nm02.pclc0.merkle.local:
> 2181,hqhd02ed01.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=hconnection-0x43e65f2a0x0, quorum=hqhd02nm01.pclc0.
> merkle.local:2181,hqhd02nm02.pclc0.merkle.local:2181,
> hqhd02ed01.pclc0.merkle.local:2181, baseZNode=/hbase-secure
> 2016-09-07 15:45:09,082 INFO
> [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)]
> org.apache.zookeeper.ClientCnxn: Opening socket connection to server
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
>
>
> Access the Q2 2016 Digital Marketing Report for a fresh set of trends
> and benchmarks in digital
> marketing<http://www2.merkleinc.com/l/47252/2016-07-
> 26/47gt7c>
>
> Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation”
> <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c>
>
> This email and any attachments transmitted with it are intended for
> use by the intended recipient(s) only. If you have received this email
> in error, please notify the sender immediately and then delete it. If
> you are not the intended recipient, you must not keep, use, disclose,
> copy or distribute this email without the author’s prior permission.
> We take precautions to minimize the risk of transmitting software
> viruses, but we advise you to perform your own virus checks on any
> attachment to this message. We cannot accept liability for any loss or
> damage caused by software viruses. The information contained in this
> communication may be confidential and may be subject to the attorney-client privilege.
>
Access the Q2 2016 Digital Marketing Report for a fresh set of trends and benchmarks in digital
marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c>

Download our latest report titled “The Case for Change: Exploring the Myths of Customer-Centric
Transformation” <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c>
This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.
Mime
View raw message