incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharad Agarwal <sharad.apa...@gmail.com>
Subject Re: random access performance of messages.
Date Fri, 21 Oct 2011 16:47:03 GMT
kafka is more suited for sequential message reads. Not really meant for
random message lookups.

Also using kafka as *long* term message store is not a good usecase.

On Fri, Oct 21, 2011 at 9:32 PM, <marko@modelcitizen.com> wrote:

> I would like to use Kafka to process messages that need to be immutably
> stored for a N-days, and during that period the msgs need to be indexed,
> searched, as well as retrieval of msg data that is queried.
>
>
>
> One approach is to read messages from Kafka and store the messages in a
> secondary db for query and data retrieval.  Once the messages are read and
> processed into the secondary db, then the messages can be discarded from
> the
> Kafka queue.
>
>
>
> Another approach is to read the messages, build an external index for
> searching that directly references the message data by Kafka-key in the
> Kafka queue itself.  In this case the Kafka becomes the message store for
> the life of the message/data.
>
>
>
> The latter would be ideal for me if the performance of query-by-key and
> message data retrieval is very good.
>
>
>
> Is random query of message+data good for Kafka?  Is this an appropriate
> usecase for Kafka?
>
>
>
> Thank you.
>
>
>
> Marko.
>
> .
>
>
>
>


-- 
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
*Disclaimer: Opinions expressed here are my own and do not represent past or
present employers.*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message