Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4FB570A3.4070404@groovie.org>
Date: Thu, 17 May 2012 14:41:55 -0700
From: Ben Bangert <ben@groovie.org>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:12.0) Gecko/20120428 Thunderbird/12.0.1
MIME-Version: 1.0
To: user@zookeeper.apache.org
Subject: Re: The State of Python Zookeeper libraries and collaboration
References: <4FB44B3B.7090604@groovie.org>
 <CAOYeBnOpe0f3-pehndeNuj2bGhcKsm9Vcgn3mYaAPmmnkbHzQA@mail.gmail.com>
In-Reply-To: 
 <CAOYeBnOpe0f3-pehndeNuj2bGhcKsm9Vcgn3mYaAPmmnkbHzQA@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature";
 boundary="------------enig05EB4AB1DD88BA338C9DBEE4"

--------------enig05EB4AB1DD88BA338C9DBEE4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 5/17/12 9:24 AM, Mark Gius wrote:
> Are you planning on writing a pure-python client (does not call out to =
the
> C bindings via zkpython) or are you planning on writing a solid wrapper=

> around the C bindings. Implementing a pure-python client would go a lon=
g
> way towards making various green thread frameworks work without having =
to
> jump through hoops.  I think we'd have to add support to Jute so that i=
t
> would generate python data classes kind of like it does now with Java a=
nd C.
>=20
> Assuming you go with a wrapper around the C bindings, I would suggest y=
ou
> take a look at something called "xthread.py", which was a thread
> synchronization primitives library that a guy proposed to the eventlet
> project a while back and provides Lock, Notify, etc etc which are safe =
to
> use and notify between real threads and green threads.  It gives a safe=
 way
> to deal with sending data and doing proper locks without having to worr=
y
> about calling out to "green" things from within non greened contexts (s=
uch
> as the zk callback functions).  It's eventlet specific, but the concept=
s
> and probably fair amount of the code can probably be adapted and use.
>=20
> Or pure-python.  That works too. :D

I looked at writing a base zookeeper replacement that includes the
higher level API's and utilizes ctypes to talk to an install zookeeper C
binding rather than using the Python C binding. This has the advantage
of working in PyPy, however it was quite a pain and probably has some
slower performance than the Python C binding. This would be pure Python,
but not quite in the way you're referring to as its not talking directly
to Zookeeper using pure Python, but still using the C API.

I'm mainly looking at having a higher level API that makes it easier to
use Zookeeper in a less error-prone manner. Like Netflix's Curator, only
with a Pythonic API since their API is fairly heavily grounded in Java
limitations. So it'll have convenience methods, a consistent API that's
usable under greenlets or threaded code, all the recipes, and well
tested and documented.

Lots of things using Zookeeper make the notion (for right or wrong) that
watch events are executed sequentially (the C API does this for example
as does the Java one AFAIK).

To handle this my plan when using greenlets was to immediately spawn a
greenlet watch processor during the ZK client initialization that would
work off a normal non-gevent-patched Queue object, and the callbacks
will drop a lambda onto the queue from the ZK thread. This ensures even
in an async environment that by default all watch events are processed
in the same order the ZK client receives them (a watch func could of
course spawn a greenlet for itself, but at that point its already safely
in the 'green context').

The Kazoo author (David LaBissoniere) has written a small test script
that verifies this approach appears to work. I'll want to test it under
heavy load of course but it seems like a rather safe and sane approach.
It also avoids a lot of the hairier pipe code that tries and shuttles
things safely back and forth.

If someone comes up with a pure Python client to Zookeeper, I'd be happy
to work on supporting that as well but its a bit beyond the level of
direct involvement I can provide.

--=20
Ben Bangert
(ben@ || http://) groovie.org


--------------enig05EB4AB1DD88BA338C9DBEE4
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPtXCqAAoJEMyDQib10m3HrqgQAI9ccv0o9Jiw7MR0WPNeXk1B
FRVgCqcYbTRlnVF4V6uC1Gn3dzMV3V9xlUVMee0tmpHgCv2gBzSbcT4YusUYEoHt
bIfkL7gVnTPgIuPTewK4jr7V6vKEBVZDrGPS0O5jdrS7HWR8cIwwQjp3ilTK6Ktq
FtSX38mt2lETqxUwxFzpn99WXB7il1KoCEWDhvyzfmt9SlmQzWpsmgIMPzIj07IY
wenMznyIrtiFUuVnwjLxZ/clzN/sYEbCTElKjUICa4p3KA4BQEGehTbaVFBhPCn5
yYJEo59C1NGaWLPr+rXWrfj4LC4w2NO00tsPdIrpHPNeFs3O+IcpXg9xA7zbIFNo
sPkibnmd+w8Ueh4hKQYTO6dmE14NDkWe0fHnn2QmC6UaUZakaftZgeK74EwC77E4
zjVvU6p7fp+Ji8dmHEmLmDKIMCeQMpIa/woiRI4MsCg+3PnxLgokIT6zWwsjqbGZ
ON1nhpNZfvg77hmgjdogi959YKWIT0Z+THlM3AWQ5iAjtoT4XyajT9UOxKUlt4FJ
WGTiypdlKkO7rNt6NAhiPh4pJFMPbpwN0reLEvr5Elq8L79NIjmgiwXLnciif3iM
8O5t4KJATkCet5nZgk5RkxFdHhmsrcqVehF+Iz6CJLYc/5v1rAjfVG3Y3zneb0iW
WWJ7ST3OYIHMZ0Kph8cK
=+KTX
-----END PGP SIGNATURE-----

--------------enig05EB4AB1DD88BA338C9DBEE4--