kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SenthilKumar K <senthilec...@gmail.com>
Subject Re: Efficient way of Searching Messages By Timestamp - Kafka
Date Sun, 28 May 2017 16:28:36 GMT
Hi Dev, It would be great if anybody share your experience on Search
Message by Timestamp ..

Cheer's,
Senthil

On May 28, 2017 2:08 AM, "SenthilKumar K" <senthilec566@gmail.com> wrote:

> Hi Team , Any help here Pls ?
>
> Cheers,
> Senthil
>
> On Sat, May 27, 2017 at 8:25 PM, SenthilKumar K <senthilec566@gmail.com>
> wrote:
>
>> Hello Kafka Developers , Users ,
>>
>>     We are exploring the SearchMessageByTimestamp feature in Kafka for
>> our use case .
>>
>>     Use Case : Kafka will be realtime message bus , users should be able
>> to pull Logs by specifying start_date and end_date or  Pull me last five
>> minutes data etc ...
>>
>> I did POC on SearchMessageByTimestamp , here is the code
>> https://gist.github.com/senthilec566/16e8e28b32834666fea132afc3a4e2f9 .
>> And i observed that Searching Messages is slow ..
>>
>> Here is small test i did :
>> Query :Fetch Logs of Last *5 minutes*:
>> Result:
>> No of Records fetched : *30*
>> Fetch Time *6210* ms
>>
>> Above test performed in a topic which has 4 partitions. In each partition
>> search & query processing happened .. in other words
>> consumer.offsetsForTimes()
>> consumer.assign(Arrays.asList(partition))
>> consumer.seek(this.partition, offsetTimestamp.offset())
>> consumer.poll(100)
>>
>> are the API calls of each partition.. I realized that , this was the
>> reason for Kafka taking more time..
>>
>> What is efficient way of implementing SerachMessageByTimeStamp ?  Is
>> Kafka right candidate for our Use Case ?
>>
>> Pls add your thoughts here ...
>>
>>
>> Cheers,
>> Senthil
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message