incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <>
Subject Kafka is live in prod @ 100%
Date Tue, 06 Dec 2011 16:30:37 GMT
I've already mentioned this before, but I wanted to give a quick shout to
let you guys know that our newest game, Deckadence, is 100% live as of

Check it out at

A little about our use case:

   - Deckadence is a game of buying and selling - or rather trading -
   cards.  Every user on Tagged owns a card.  There are 100M uses on Tagged,
   so that means there are 100M cards to trade.
   - Kafka enables real-time delivery of events in the game
   - An end user browser makes a long-poll event http connection to receive
   1:1 messages and 1:M messages from a specialized http server we built for
   this purpose.  1:M messages are delivered from Kafka.
   - Because of this design, we can publish a message anywhere inside our
   datacenter and send it directly and immediately to any other system that is
   subscribed to Kafka, or to an end-user browser
   - Every update event for every card is sent to a unique topic that
   represents the users card.
   - When a user is browsing any card or list of cards - say a search
   result - their browser subscribes to all of the cards on screen.
   - The effect of this is that any changes to any card seen on-screen are
   seen in real-time by all users of the game
   - Our primary producers and consumers are PHP and NodeJS, respectively

Well, I plan to write up more about this use case in the near future.  As
you might have guessed, this is just about as far away from the original
intent of Kafka as you could get - we have PHP that sends messages to
Kafka.  Since it's not good to hold a TCP connection open in PHP, we had to
do some trickery here.  There was no existing Node client so we had to
write our own.  And since there are 100 million users registered on Tagged,
that means we could have in theory 100M topics.  Of course in practice we
have far fewer than that.  One of the main things we currently have to do
is aggressively clean topics.  But basically we have many topics, few
messages (relatively) per topic.  And order matters, so we had to deal with
ensuring that we could handle the number of topics we would create, and
ensure ordered delivery and receipt.

In the future I have big plans for Kafka, another feature is currently in
private test and will be released to the public soon (it uses Kafka in a
more traditional way).  And we hope to have many more in 2012...

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message