incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <tgaut...@tagged.com>
Subject Re: Kafka is live in prod @ 100%
Date Sun, 11 Dec 2011 15:56:04 GMT
I didn't. Topics are restricted to one broker one partition.



On Dec 11, 2011, at 2:57 AM, Inder Pall <inder.pall@gmail.com> wrote:

> taylor
>
> great to hear this. how did you achieve ordering without restricting a
> topic to one broker onw partiton.
> On Dec 6, 2011 10:01 PM, "Taylor Gautier" <tgautier@tagged.com> wrote:
>
>> I've already mentioned this before, but I wanted to give a quick shout to
>> let you guys know that our newest game, Deckadence, is 100% live as of
>> yesterday.
>>
>> Check it out at http://www.tagged.com/deckadence.html
>>
>> A little about our use case:
>>
>>  - Deckadence is a game of buying and selling - or rather trading -
>>  cards.  Every user on Tagged owns a card.  There are 100M uses on Tagged,
>>  so that means there are 100M cards to trade.
>>  - Kafka enables real-time delivery of events in the game
>>  - An end user browser makes a long-poll event http connection to receive
>>  1:1 messages and 1:M messages from a specialized http server we built for
>>  this purpose.  1:M messages are delivered from Kafka.
>>  - Because of this design, we can publish a message anywhere inside our
>>  datacenter and send it directly and immediately to any other system that
>> is
>>  subscribed to Kafka, or to an end-user browser
>>  - Every update event for every card is sent to a unique topic that
>>  represents the users card.
>>  - When a user is browsing any card or list of cards - say a search
>>  result - their browser subscribes to all of the cards on screen.
>>  - The effect of this is that any changes to any card seen on-screen are
>>  seen in real-time by all users of the game
>>  - Our primary producers and consumers are PHP and NodeJS, respectively
>>
>> Well, I plan to write up more about this use case in the near future.  As
>> you might have guessed, this is just about as far away from the original
>> intent of Kafka as you could get - we have PHP that sends messages to
>> Kafka.  Since it's not good to hold a TCP connection open in PHP, we had to
>> do some trickery here.  There was no existing Node client so we had to
>> write our own.  And since there are 100 million users registered on Tagged,
>> that means we could have in theory 100M topics.  Of course in practice we
>> have far fewer than that.  One of the main things we currently have to do
>> is aggressively clean topics.  But basically we have many topics, few
>> messages (relatively) per topic.  And order matters, so we had to deal with
>> ensuring that we could handle the number of topics we would create, and
>> ensure ordered delivery and receipt.
>>
>> In the future I have big plans for Kafka, another feature is currently in
>> private test and will be released to the public soon (it uses Kafka in a
>> more traditional way).  And we hope to have many more in 2012...
>>

Mime
View raw message