Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of james_mulcahy@apple.com
 designates 17.72.148.12 as permitted sender)
Content-type: text/plain; charset=windows-1252
MIME-version: 1.0 (Mac OS X Mail 7.3 \(1878.2\))
Subject: Re: Unbalanced client connections
From: James Mulcahy <james_mulcahy@apple.com>
In-reply-to: <1404479783.33959.YahooMailNeo@web142305.mail.bf1.yahoo.com>
Date: Fri, 04 Jul 2014 14:36:26 +0100
Content-transfer-encoding: quoted-printable
Message-id: <E70DB2CA-34BD-4636-9756-51319B259CE2@apple.com>
References: <8801B9CA-6A0A-4249-BB37-A79C4534FAB8@apple.com>
 <1404478660.75055.YahooMailNeo@web142301.mail.bf1.yahoo.com>
 <2BDA69A7-AF21-4EDD-B2EE-E1258C471645@apple.com>
 <1404479783.33959.YahooMailNeo@web142305.mail.bf1.yahoo.com>
To: user@zookeeper.apache.org, Flavio Junqueira <fpjunqueira@yahoo.com>

On 4 Jul 2014, at 14:16, Flavio Junqueira =
<fpjunqueira@yahoo.com.INVALID> wrote:

> Ok, so a couple of obvious checks

Sure=85

> - Are you passing a connection string with all five servers?

Yes, most definitely.  Prior to deployment of this I did some extensive =
testing where I killed off ZK servers randomly to test our clients=92 =
ability to reconnect on to another server in the cluster.  I know that =
if they absolutely need to, they can connect elsewhere =97 but the =
graphs show they almost always pick the same server.

> - Are you calling zoo_deterministic_conn_order(1) by any chance (you =
shouldn't if you want shuffling)?

No, I wasn=92t aware of that function =97 but in mentioning it you=92ve =
led me to the code that does the shuffling.  Is there anything on the =
server side to force a client to move elsewhere if the server has a =
disproportional number of the clients connected to it?  That=92s the =
function I though I had read exists?  That said, given a sufficiently =
random random() function, it looks like the permute should do enough to =
stop all clients arriving on the same server initially anyway.

Perhaps I=92ll need to add some instrumentation dump out the permuted =
connection list and see how it varies across the clients?

=97James

> -Flavio
>=20
>=20
> On Friday, July 4, 2014 2:01 PM, James Mulcahy =
<james_mulcahy@apple.com> wrote:
>=20
>=20
>>=20
>>=20
>>=20
>> Hi Flavio,
>>=20
>> Thanks for the quick response =97 and apologies for not including =
these details up front!
>>=20
>> - C client binding
>> - 99.99% MacOS X Clients (10.9.2), with a couple of Linux Clients =
(Ubuntu 14.04)
>> - All ZK nodes are Linux (Ubuntu 14.404)
>> - ZooKeeper 3.4.6
>>=20
>> No Windows involved here=85.
>>=20
>> =97James
>>=20
>>=20
>> On 4 Jul 2014, at 13:57, Flavio Junqueira =
<fpjunqueira@yahoo.com.INVALID> wrote:
>>=20
>>> Hi James,
>>>=20
>>> Are you using the C or the Java client binding? What's the OS? I'm =
asking because there is an issue with the randomization of the connect =
string on Windows we found, but I haven't created a jira for it yet.
>>>=20
>>> -Flavio=20
>>>=20
>>>=20
>>> On Friday, July 4, 2014 10:41 AM, James Mulcahy =
<james_mulcahy@apple.com> wrote:
>>>=20
>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> Hello,
>>>>=20
>>>> I run a 5 node ZooKeeper ensemble, with ~900 clients connected at a =
given time.  I=92m noticing that at any one point in time, all the =
clients are generally connected to the same ZooKeeper node.
>>>>=20
>>>> Looking back over the graphs I have which track this, there has =
only been one brief period where one node didn=92t have >90% of the =
clients; and during that period, two nodes shared roughly 50% of the =
clients each.
>>>>=20
>>>> Is this expected behaviour?  Is there anything I can do to tune =
this, to encourage the clients to be more balanced?
>>>>=20
>>>> My expectation was that the clients would self-balance =97 I =
thought I=92d read that somewhere in the documentation, but I can=92t =
find a reference for that now.
>>>>=20
>>>> Thanks in advance,
>>>>=20
>>>> =97James
>>>>=20
>>=20
>>=20