hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Me <...@beobal.com>
Subject Re: Sequence Number Generation With Zookeeper
Date Sat, 07 Aug 2010 14:08:24 GMT
Hi all,

we have something implementing the optimistic concurrency approach to
sequence generation that we've been running in production for some time now.
We don't see a huge amount of contention over the sequence counters as the
nature of our app lends itself well to partitioned keys. Initially, we coded
up the simplest thing we thought could work and deployed it, figuring that
we'd have plenty of scope for improvement once we saw it running with real
load. However, to date its been ticking over so well we've not really had
cause to spend any further effort on it.

There's plenty of scope for improvement though, two of the things we had
thought we would need to do sooner rather than later are implement an
exponential backoff scheme (like Ted describes) when there is contention
over a given counter, and to add a more performant network interface than
HTTP. Like I say though, this just hasn't been a high enough priority for us
yet.

Anyway, we've been meaning to open source this for a while now, and prompted
by this thread, I just spent an afternoon tidying up a little and pushing to
github. Its at http://github.com/talisplatform/H1  and any feedback would be
gratefully received.

Cheers,
Sam

On 7 August 2010 03:40, Ted Dunning <ted.dunning@gmail.com> wrote:

> Tell him that we will all look over your code so he gets immediate free
> consulting.
>
> On Fri, Aug 6, 2010 at 7:39 PM, David Rosenstrauch <darose@darose.net
> >wrote:
>
> > I'll run it by my boss next week.
> >
> > DR
> >
> >
> > On 08/06/2010 07:30 PM, Mahadev Konar wrote:
> >
> >> Hi David,
> >>  I think it would be really useful. It would be very helpful for someone
> >> looking for geenrating unique tokens/generations ids ( I can think of
> >> plenty
> >> of applications for this).
> >>
> >> Please do consider contributing it back to the community!
> >>
> >> Thanks
> >> mahadev
> >>
> >>
> >> On 8/6/10 7:10 AM, "David Rosenstrauch"<darose@darose.net>  wrote:
> >>
> >>  Perhaps.  I'd have to ask my boss for permission to release the code.
> >>>
> >>> Is this something that would be interesting/useful to other people?  If
> >>> so, I can ask about it.
> >>>
> >>> DR
> >>>
> >>> On 08/05/2010 11:02 PM, Jonathan Holloway wrote:
> >>>
> >>>> Hi David,
> >>>>
> >>>> We did discuss potentially doing this as well.  It would be nice to
> get
> >>>> some
> >>>> recipes for Zookeeper done for this area, if people think it's useful.
> >>>>  Were
> >>>> you thinking of submitting this back as a recipe, if not then I could
> >>>> potentially work on such a recipe instead.
> >>>>
> >>>> Many thanks,
> >>>> Jon.
> >>>>
> >>>>
> >>>>  I just ran into this exact situation, and handled it like so:
> >>>>>
> >>>>> I wrote a library that uses the option (b) you described above.
 Only
> >>>>> instead of requesting a single sequence number, you request a block
> of
> >>>>> them
> >>>>> at a time from Zookeeper, and then locally use them up one by one
> from
> >>>>> the
> >>>>> block you retrieved.  Retrieving by block (e.g., by blocks of 10000
> at
> >>>>> a
> >>>>> time) eliminates the contention issue.
> >>>>>
> >>>>> Then, if you're finished assigning ID's from that block, but still
> have
> >>>>> a
> >>>>> bunch of ID's left in the block, the library has another function
to
> >>>>> "push
> >>>>> back" the unused ID's.  They'll then get pulled again in the next
> block
> >>>>> retrieval.
> >>>>>
> >>>>> We don't actually have this code running in production yet, so I
> can't
> >>>>> vouch for how well it works.  But the design was reviewed and given
> the
> >>>>> thumbs up by the core developers on the team, and the implementation
> >>>>> passes
> >>>>> all my unit tests.
> >>>>>
> >>>>> HTH.  Feel free to email back with specific questions if you'd like
> >>>>> more
> >>>>> details.
> >>>>>
> >>>>> DR
> >>>>>
> >>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message