zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Announcement - ZK IDGenerator code released to open source (finally!) :-)
Date Thu, 19 May 2011 17:56:57 GMT
Nice.

Some nits:

a) why sourceforge and not github?  SF makes it harder to browse your code
and harder to fork the code for new development because it is SVN based.

b) it would be handy for your design document to be in PDF as well as ODT
format.  Not everybody has open office.

c) most importantly, SF lists your project as being GPL, but the license in
the source code is ASL.  Which is it?

On Thu, May 19, 2011 at 10:43 AM, David Rosenstrauch <darose@darose.net>wrote:

> I've been planning to do this for a long time (see messages below), but
> finally just got around to this.  The IDGenerator code that I've discussed
> on the list a bunch of times has now been released as open source.  You can
> find it on sourceforge at http://sourceforge.net/projects/zkidgen/ .
>  (Sorry - it was just easier for me to release it via SF/svn, than
> github/git.  If anyone wants to move it over to github please feel free.)
>
> We've been using the IDGenerator in a production environment, and it's been
> working great - haven't had to touch the code in months.  It currently
> generates several hundred thousand new unique ID's per Hadoop job, with each
> job running several hundred simultaneous reducers.   The app is able to
> handle this level of load / contention without breaking a sweat.
>
> There's a design doc included with the project source code explaining a bit
> about its rationale, its architecture, and usage.  But please feel free to
> email back with any additional questions, feedback, bug reports, etc.  (And
> hopefully bug *fixes* too!)  :-)
>
> Hope this is of use to the ZK community.  I'd love to hear back from anyone
> who finds it helpful!
>
> Best,
>
> DR
>
> -------- Original Message --------
> Subject: Re: Zookeeper for generating sequential IDs
> Date: Tue, 08 Mar 2011 13:13:53 -0500
> From: David Rosenstrauch <darose@darose.net>
> Reply-To: user@zookeeper.apache.org
> To: user@zookeeper.apache.org
>
> On 03/08/2011 01:09 PM, David Rosenstrauch wrote:
>
>> On 02/25/2011 04:50 AM, Ertio Lew wrote:
>>
>>> Hi all,
>>>
>>> I am involved in a project where we're building a social application
>>> using Cassandra DB and Java. I am looking for a solution to generate
>>> unique sequential IDs for the content on the application. I have been
>>> suggested by some people to have a look to Zookeeper for this. I
>>> would highly appreciate if anyone can suggest if zookeeper is suitable
>>> for this purpose and any good resources to gain information about
>>> zookeeper.
>>>
>>> Since the application is based on a eventually consistent distributed
>>> platform using Cassandra, we have felt a need to look over to other
>>> solutions instead of building our own using our DB.
>>>
>>> Any kind of comments, suggestions are highly welcomed! :)
>>>
>>> Regards
>>> Ertio Lew.
>>>
>>
>> I ran into a similar id-generation issue, and wrote a library for it.
>> (Details described in this msg:
>>
>> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201008.mbox/%3C4C5B7656.4020200@darose.net%3E
>> .)
>>
>> Been planning to release it to the community, but haven't gotten around
>> to it yet.
>>
>> Not sure my solution is exactly what you're looking for though.
>>
>> HTH,
>>
>> DR
>>
>
> BTW, that email is old.  We now have had this running quite reliably in
> production for several months now.  It's being used by M/R jobs running
> 100 simultaneous reducers, each accessing the ID generator, and
> assigning nearly 1 million ID's per job in total.
>
> DR
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message