Kafka is relatively stable and has a active well-supported news-group as well.
As discussed by Brian, you would be inverting the paradigm of store-process. Essentially in your original approach, you are storing the messages first and then processing them after the fact. In the Kafka model, you would process the messages as they come in.
Since you are thinking about parallelism anyways, I trust that your processing paradigm is inherently paralleizable.
Brian -- thanks.
This might be off-topic, so I'll keep it short. Kafka is reasonably stable? Mature (I see it's in the Incubator)? Relative to Cassandra?
> We were looking to do the same thing, but in the end decided
> to go with Kafka.
> Given your throughput requirements, Kafka might be a good
> option for you as well.