incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Kafka is live in prod @ 100%
Date Sun, 11 Dec 2011 10:56:51 GMT
taylor

great to hear this. how did you achieve ordering without restricting a
topic to one broker onw partiton.
On Dec 6, 2011 10:01 PM, "Taylor Gautier" <tgautier@tagged.com> wrote:

> I've already mentioned this before, but I wanted to give a quick shout to
> let you guys know that our newest game, Deckadence, is 100% live as of
> yesterday.
>
> Check it out at http://www.tagged.com/deckadence.html
>
> A little about our use case:
>
>   - Deckadence is a game of buying and selling - or rather trading -
>   cards.  Every user on Tagged owns a card.  There are 100M uses on Tagged,
>   so that means there are 100M cards to trade.
>   - Kafka enables real-time delivery of events in the game
>   - An end user browser makes a long-poll event http connection to receive
>   1:1 messages and 1:M messages from a specialized http server we built for
>   this purpose.  1:M messages are delivered from Kafka.
>   - Because of this design, we can publish a message anywhere inside our
>   datacenter and send it directly and immediately to any other system that
> is
>   subscribed to Kafka, or to an end-user browser
>   - Every update event for every card is sent to a unique topic that
>   represents the users card.
>   - When a user is browsing any card or list of cards - say a search
>   result - their browser subscribes to all of the cards on screen.
>   - The effect of this is that any changes to any card seen on-screen are
>   seen in real-time by all users of the game
>   - Our primary producers and consumers are PHP and NodeJS, respectively
>
> Well, I plan to write up more about this use case in the near future.  As
> you might have guessed, this is just about as far away from the original
> intent of Kafka as you could get - we have PHP that sends messages to
> Kafka.  Since it's not good to hold a TCP connection open in PHP, we had to
> do some trickery here.  There was no existing Node client so we had to
> write our own.  And since there are 100 million users registered on Tagged,
> that means we could have in theory 100M topics.  Of course in practice we
> have far fewer than that.  One of the main things we currently have to do
> is aggressively clean topics.  But basically we have many topics, few
> messages (relatively) per topic.  And order matters, so we had to deal with
> ensuring that we could handle the number of topics we would create, and
> ensure ordered delivery and receipt.
>
> In the future I have big plans for Kafka, another feature is currently in
> private test and will be released to the public soon (it uses Kafka in a
> more traditional way).  And we hope to have many more in 2012...
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message