storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Cooper <>
Subject RE: Replacing DRPC With Kafka
Date Fri, 04 Apr 2014 09:11:40 GMT
We use kafka for inter-topology communication. Going from input -> kafka -> storm ->
kafka -> output takes around 20-40ms. Although you need to use kafka 0.8 or greater, as
previous versions force an fsync on every message, which makes performance plummet.

The main reasons we went for kafka rather than DRPC was that kafka gives us a permanent log
of all the messages in and out of the system, and that messages won't be lost on a server
failure (whereas they might be with DRPC, as that just uses an in-memory queue)


From: Jon Logan []
Sent: 04 April 2014 02:44
Subject: Replacing DRPC With Kafka

Has anyone attempted to replace Storm DRPC with Kafka?  My main concern stems from the weight
of Kafka topics...especially for the handling of return results to clients.

View raw message