Mailing-List: contact kafka-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-users@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of sharad.apache@gmail.com
 designates 209.85.214.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <004801cc900a$e6ca6bc0$b45f4340$@modelcitizen.com>
References: <004801cc900a$e6ca6bc0$b45f4340$@modelcitizen.com>
Date: Fri, 21 Oct 2011 22:17:03 +0530
Message-ID: 
 <CAPqfsdxFQ7tMBqvPvanQAPR2=BQq4R2UoKZ_+qMsE4FabESsXA@mail.gmail.com>
Subject: Re: random access performance of messages.
From: Sharad Agarwal <sharad.apache@gmail.com>
To: kafka-users@incubator.apache.org
Content-Type: multipart/alternative; boundary=0015175885daa277f604afd1d204

--0015175885daa277f604afd1d204
Content-Type: text/plain; charset=ISO-8859-1

kafka is more suited for sequential message reads. Not really meant for
random message lookups.

Also using kafka as *long* term message store is not a good usecase.

On Fri, Oct 21, 2011 at 9:32 PM, <marko@modelcitizen.com> wrote:

> I would like to use Kafka to process messages that need to be immutably
> stored for a N-days, and during that period the msgs need to be indexed,
> searched, as well as retrieval of msg data that is queried.
>
>
>
> One approach is to read messages from Kafka and store the messages in a
> secondary db for query and data retrieval.  Once the messages are read and
> processed into the secondary db, then the messages can be discarded from
> the
> Kafka queue.
>
>
>
> Another approach is to read the messages, build an external index for
> searching that directly references the message data by Kafka-key in the
> Kafka queue itself.  In this case the Kafka becomes the message store for
> the life of the message/data.
>
>
>
> The latter would be ideal for me if the performance of query-by-key and
> message data retrieval is very good.
>
>
>
> Is random query of message+data good for Kafka?  Is this an appropriate
> usecase for Kafka?
>
>
>
> Thank you.
>
>
>
> Marko.
>
> .
>
>
>
>


-- 
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
*Disclaimer: Opinions expressed here are my own and do not represent past or
present employers.*

--0015175885daa277f604afd1d204--