zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Tunnicliffe <...@beobal.com>
Subject Re: Sequential Id generation with Zookeeper - Tips to improve performance
Date Wed, 13 Jun 2012 11:23:16 GMT
Yes, the contention occurs in a multithreaded context when two or more
clients attempt to increment the same counter concurrently. The first
in wins and the following updates are rejected as the znode version no
longer matches the one obtained from the getData call. A clients who's
increment is rejected just goes around the loop again, we don't have
any more sophisticated logic in place to ensure fairness.

We don't do any special tuning on the server side and we're running in
AWS on standard m1.large instances. It could be the case that we're
only seeing 10s of updates/s per unique counter like you, but because
our counters are partitioned and we have multiple concurrent clients,
total throughput is increased.

Cheers,
Sam

On 13 June 2012 11:52, Charith Wickramarachchi
<charith.dhanushka@gmail.com> wrote:
> Thanks Sam for the reply.
>
> The contention happens when we to setData from muliple clients right ? (due
> to locking). But here even when i try using a single client the throughput
> is in 20-30 id/s range.
> Did you do any tuning in the server side. ?
>
> regards,
> Charith
>
> On Wed, Jun 13, 2012 at 6:17 AM, Sam Tunnicliffe <sam@beobal.com> wrote:
>
>> Hi Charith,
>>
>> can you partition your counters somehow, rather than having a single
>> global counter? This would let you use multiple znodes, reducing
>> contention during the setData call.
>> We use almost exactly the same approach as you in our zookeeper backed
>> sequence server[1] but with sharded counters and haven't had any
>> issues with throughput (sorry, I don't have perf numbers to hand, but
>> peak load pushes a few hundred r/s through)
>>
>> Cheers,
>> Sam
>>
>> [1] https://github.com/talis/H1
>>
>> On 13 June 2012 05:20, Charith Wickramarachchi
>> <charith.dhanushka@gmail.com> wrote:
>> > Hi All ,
>> >
>> > I'm evaluating Apache Zookeepers to use in my project where i have a
>> > requirement to generate unique and sequential message ids(long) across a
>> > cluster.
>> > After some  web search  i found[1] and some other different suggestions.
>> I
>> > started with the solution suggested in [1] but even with single client  i
>> > only got a throughput around 20-30 ids/sec. But according to the
>> performance
>> > numbers pointed in the website i think i should be able get better
>> results (
>> > i m expecting to get at least 1000 id/s as i m using this for a MoM where
>> > for each message i'll be calling zk id generator.)
>> >
>> > In this case I m using ZK version 3.3.4 running in my local machine with
>> > default settings and client also running in the same machine. I m
>> attaching
>> > my id generator class.
>> >
>> > Any tips to improve the performance of this is highly appreciated. Am i
>> > doing some thing wrong ?
>> >
>> >
>> > [1]
>> http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
>> >
>> > cheers,
>> > Charith
>> > --
>> > Charith Dhanushka Wickramarachchi
>> > http://charithwiki.blogspot.com/
>> >
>>
>
>
>
> --
> Charith Dhanushka Wickramarachchi
> http://charithwiki.blogspot.com/

Mime
View raw message