hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: Sequence Number Generation With Zookeeper
Date Sat, 07 Aug 2010 02:39:37 GMT
I'll run it by my boss next week.

DR

On 08/06/2010 07:30 PM, Mahadev Konar wrote:
> Hi David,
>   I think it would be really useful. It would be very helpful for someone
> looking for geenrating unique tokens/generations ids ( I can think of plenty
> of applications for this).
>
> Please do consider contributing it back to the community!
>
> Thanks
> mahadev
>
>
> On 8/6/10 7:10 AM, "David Rosenstrauch"<darose@darose.net>  wrote:
>
>> Perhaps.  I'd have to ask my boss for permission to release the code.
>>
>> Is this something that would be interesting/useful to other people?  If
>> so, I can ask about it.
>>
>> DR
>>
>> On 08/05/2010 11:02 PM, Jonathan Holloway wrote:
>>> Hi David,
>>>
>>> We did discuss potentially doing this as well.  It would be nice to get some
>>> recipes for Zookeeper done for this area, if people think it's useful.  Were
>>> you thinking of submitting this back as a recipe, if not then I could
>>> potentially work on such a recipe instead.
>>>
>>> Many thanks,
>>> Jon.
>>>
>>>
>>>> I just ran into this exact situation, and handled it like so:
>>>>
>>>> I wrote a library that uses the option (b) you described above.  Only
>>>> instead of requesting a single sequence number, you request a block of them
>>>> at a time from Zookeeper, and then locally use them up one by one from the
>>>> block you retrieved.  Retrieving by block (e.g., by blocks of 10000 at a
>>>> time) eliminates the contention issue.
>>>>
>>>> Then, if you're finished assigning ID's from that block, but still have a
>>>> bunch of ID's left in the block, the library has another function to "push
>>>> back" the unused ID's.  They'll then get pulled again in the next block
>>>> retrieval.
>>>>
>>>> We don't actually have this code running in production yet, so I can't
>>>> vouch for how well it works.  But the design was reviewed and given the
>>>> thumbs up by the core developers on the team, and the implementation passes
>>>> all my unit tests.
>>>>
>>>> HTH.  Feel free to email back with specific questions if you'd like more
>>>> details.
>>>>
>>>> DR

Mime
View raw message