hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: Sequence Number Generation With Zookeeper
Date Fri, 06 Aug 2010 14:10:00 GMT
Perhaps.  I'd have to ask my boss for permission to release the code.

Is this something that would be interesting/useful to other people?  If 
so, I can ask about it.


On 08/05/2010 11:02 PM, Jonathan Holloway wrote:
> Hi David,
> We did discuss potentially doing this as well.  It would be nice to get some
> recipes for Zookeeper done for this area, if people think it's useful.  Were
> you thinking of submitting this back as a recipe, if not then I could
> potentially work on such a recipe instead.
> Many thanks,
> Jon.
>> I just ran into this exact situation, and handled it like so:
>> I wrote a library that uses the option (b) you described above.  Only
>> instead of requesting a single sequence number, you request a block of them
>> at a time from Zookeeper, and then locally use them up one by one from the
>> block you retrieved.  Retrieving by block (e.g., by blocks of 10000 at a
>> time) eliminates the contention issue.
>> Then, if you're finished assigning ID's from that block, but still have a
>> bunch of ID's left in the block, the library has another function to "push
>> back" the unused ID's.  They'll then get pulled again in the next block
>> retrieval.
>> We don't actually have this code running in production yet, so I can't
>> vouch for how well it works.  But the design was reviewed and given the
>> thumbs up by the core developers on the team, and the implementation passes
>> all my unit tests.
>> HTH.  Feel free to email back with specific questions if you'd like more
>> details.
>> DR

View raw message