zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Announcement - ZK IDGenerator code released to open source (finally!) :-)
Date Thu, 19 May 2011 17:43:02 GMT
I've been planning to do this for a long time (see messages below), but 
finally just got around to this.  The IDGenerator code that I've 
discussed on the list a bunch of times has now been released as open 
source.  You can find it on sourceforge at 
http://sourceforge.net/projects/zkidgen/ .  (Sorry - it was just easier 
for me to release it via SF/svn, than github/git.  If anyone wants to 
move it over to github please feel free.)

We've been using the IDGenerator in a production environment, and it's 
been working great - haven't had to touch the code in months.  It 
currently generates several hundred thousand new unique ID's per Hadoop 
job, with each job running several hundred simultaneous reducers.   The 
app is able to handle this level of load / contention without breaking a 
sweat.

There's a design doc included with the project source code explaining a 
bit about its rationale, its architecture, and usage.  But please feel 
free to email back with any additional questions, feedback, bug reports, 
etc.  (And hopefully bug *fixes* too!)  :-)

Hope this is of use to the ZK community.  I'd love to hear back from 
anyone who finds it helpful!

Best,

DR

-------- Original Message --------
Subject: Re: Zookeeper for generating sequential IDs
Date: Tue, 08 Mar 2011 13:13:53 -0500
From: David Rosenstrauch <darose@darose.net>
Reply-To: user@zookeeper.apache.org
To: user@zookeeper.apache.org

On 03/08/2011 01:09 PM, David Rosenstrauch wrote:
> On 02/25/2011 04:50 AM, Ertio Lew wrote:
>> Hi all,
>>
>> I am involved in a project where we're building a social application
>> using Cassandra DB and Java. I am looking for a solution to generate
>> unique sequential IDs for the content on the application. I have been
>> suggested by some people to have a look to Zookeeper for this. I
>> would highly appreciate if anyone can suggest if zookeeper is suitable
>> for this purpose and any good resources to gain information about
>> zookeeper.
>>
>> Since the application is based on a eventually consistent distributed
>> platform using Cassandra, we have felt a need to look over to other
>> solutions instead of building our own using our DB.
>>
>> Any kind of comments, suggestions are highly welcomed! :)
>>
>> Regards
>> Ertio Lew.
>
> I ran into a similar id-generation issue, and wrote a library for it.
> (Details described in this msg:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201008.mbox/%3C4C5B7656.4020200@darose.net%3E
> .)
>
> Been planning to release it to the community, but haven't gotten around
> to it yet.
>
> Not sure my solution is exactly what you're looking for though.
>
> HTH,
>
> DR

BTW, that email is old.  We now have had this running quite reliably in
production for several months now.  It's being used by M/R jobs running
100 simultaneous reducers, each accessing the ID generator, and
assigning nearly 1 million ID's per job in total.

DR

Mime
View raw message