zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Kou <bitana...@gmail.com>
Subject Re: The State of Python Zookeeper libraries and collaboration
Date Sun, 20 May 2012 19:29:23 GMT
A pure Python implementation shouldn't be needed for gevent. It should be
sufficient to run the Zookeeper client in async mode in a background thread
(with mt or st.. personally I prefer st), and then use an eventfd() or a
pipe() to notify gevent.

In fact, if you look into gevent's thread pool library - they're using
libev's async signal which is also based on eventfd() and pipe(). So it's
not a new trick.

Best Regards,
Martin Kou

On Thu, May 17, 2012 at 9:24 AM, Mark Gius <mgius7096@gmail.com> wrote:

> Are you planning on writing a pure-python client (does not call out to the
> C bindings via zkpython) or are you planning on writing a solid wrapper
> around the C bindings. Implementing a pure-python client would go a long
> way towards making various green thread frameworks work without having to
> jump through hoops.  I think we'd have to add support to Jute so that it
> would generate python data classes kind of like it does now with Java and
> C.
>
> Assuming you go with a wrapper around the C bindings, I would suggest you
> take a look at something called "xthread.py", which was a thread
> synchronization primitives library that a guy proposed to the eventlet
> project a while back and provides Lock, Notify, etc etc which are safe to
> use and notify between real threads and green threads.  It gives a safe way
> to deal with sending data and doing proper locks without having to worry
> about calling out to "green" things from within non greened contexts (such
> as the zk callback functions).  It's eventlet specific, but the concepts
> and probably fair amount of the code can probably be adapted and use.
>
> Or pure-python.  That works too. :D
>
> Mark
>
> On Wed, May 16, 2012 at 5:50 PM, Ben Bangert <ben@groovie.org> wrote:
>
> > It would seem that about 6 months or ago or so, there wasn't much out
> > there in terms of higher level Python libs for Zookeeper. There was the
> > Cloudera article on queues, and txzookeeper (which I'm sure many of us
> > not using twisted immediately ignored).
> >
> > In the time since, several people including myself needed solutions
> > involving Zookeeper with Python and seeing nothing out there all
> > apparently began writing libraries (judging from the project timelines
> > in most cases). I've been collaborating with the author of zc.zk (Jim
> > Fulton) for awhile and we decided it'd make more sense to merge our
> > efforts. In this spirit I began contacting all the other developers to
> > gauge their interest and most have been interested.
> >
> > I created a python-zk organization on GitHub to be the home for this
> > effort and moved over the zc.zk library (which people apparently had a
> > hard time locating), along with the fairly widely used staticly compiled
> > Python Zookeeper binding.
> >
> > https://github.com/python-zk
> >
> > Next up is to create the new merged core which I plan on basing mostly
> > around the cleanest implementation I have seen so far (which also
> > happens to be one of the only gevent compatible ones), kazoo. I've
> > talked with the primary author of Kazoo, and the name may remain with
> > the new merged package or it may get a new name if that doesn't work.
> > I'm not terribly tied to names as much as I am to solid, well tested,
> > well documented working code... but having catchy names does seem to
> help.
> >
> > I'm currently working on this full-time, so I expect it to be in a
> > usable state in a week or so (hopefully not too optimistic). If you're
> > interested in helping out, the more the better, please feel free to
> > e-mail me directly or respond here.
> >
> > This stuff is complex, it needs many eyes on it and lots of code review.
> >
> > This hopefully explains why I'm so interested in having a single Python
> > Zookeeper library along similar caliber to Netflix's Curator that has:
> > - Very thorough unit/integration tests (100% coverage minimum)
> > - Cleanly handles connection loss
> > - Works under gevent or threaded/blocking
> > - Very well documented (API docs and narrative)
> > - Implements all the Zookeeper recipes
> > - Service Discovery/Management
> > - Higher level utility functions for common Zookeeper tasks
> >
> > In the mean-time, here is a summary of my research efforts and code
> > review (if something isn't accurate, please feel free to correct).
> >
> > Please don't take this as a critique, I'm just trying to document what
> > is out there for my own reference on merging and hopefully so other
> > people coming along don't continue to replicate this. :)
> >
> >
> > gevent-zookeeper
> >    - https://github.com/jrydberg/gevent-zookeeper/
> >
> >    - Works under gevent
> >    - No tests
> >    - No documentation
> >
> > kazoo
> >    - https://github.com/nimbusproject/kazoo
> >
> >    - Resilient Client
> >    - Basic Lock (Uses UUID properly)
> >    - Some Tests (Integrated)
> >    - No documentation (doc strings only)
> >    - Works under gevent
> >
> > pykeeper
> >    - https://github.com/nkvoll/pykeeper
> >
> >    - Higher level client (not resiliant to errors)
> >    - Documentation
> >    - Some tests (Integrated)
> >
> > txzookeeper
> >    - JuJu Team
> >    - https://launchpad.net/txzookeeper
> >
> >    - Resilient Client
> >    - Doesn't handle create node edge-case
> >    - Basic Lock (open bug filed to handle the UUID bit)
> >    - Queue, ReliableQueue, SerializedQueue
> >    - No documentation (doc strings only)
> >    - Usable only from twisted
> >    - Well tested (Integrated)
> >
> > twitter zookeeper lib
> >    -
> >
> >
> https://github.com/twitter/commons/tree/master/src/python/twitter/common/zookeeper
> >
> >    - Resilient Client
> >    - Handles create node edge-case
> >    - Service Registration/Discovery
> >    - Some documentation
> >    - Well tested (Integrated)
> >    - Tied to a lot of twitter commons code
> >
> > zkpython (improvements to a fork of the official bindings)
> >    - https://github.com/duncf/zkpython/
> >
> >    - Resilient Client
> >    - Basic Lock (Using unique id rather than UUID)
> >    - Handles create node edge-case
> >    - Some Tests (Integrated)
> >    - No additional docs
> >
> > zc.zk
> >    - https://github.com/python-zk/zc.zk
> >
> >    - Non-resilient Client (reconnects must be handled)
> >    - Higher level automatic watch functionality
> >    - Service Registration/Discovery
> >    - Well tested (Unit and Integration tests)
> >    - Documented (on usage, source code is missing doc strings)
> >
> > zktools
> >    - https://github.com/mozilla-services/zktools
> >
> >    - Relies on zc.zk
> >    - Shared Read/Write Locks
> >    - AsyncLock
> >    - Revokable Locks
> >    - Tests (Integrated)
> >
> > zoop
> >    - https://github.com/davidmiller/zoop
> >
> >    - Doesn't handle create node edge-case
> >    - Doesn't handle retryable exceptions
> >    - Revokable Lock (Doesn't handle create node edge-case, uses a
> permanent
> >                      node instead of ephemeral)
> >    - Tested (Unit tests via ZK mocks)
> >    - Well Documented (doc strings and narrative docs)
> >
> >
> > --
> > Ben Bangert
> > (ben@ || http://) groovie.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message