distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leigh Stewart <lstew...@twitter.com.INVALID>
Subject Re: Use DL stream to store offsets?
Date Wed, 02 Nov 2016 16:19:37 GMT
Whats your timeline Khurrum? Maybe we can work something out.

On Wed, Nov 2, 2016 at 8:15 AM, Leigh Stewart <lstewart@twitter.com> wrote:

> We have in fact built something like this. No plans as yet to release, but
> I think we would like to eventually.
> On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnasimm@gmail.com>
> wrote:
>> As part of implementing the kafka subscriber interface, I am wondering is
>> there anyone use DL stream for storing the offsets?
>> For example, if I have N streams (0..N-1), I need to track the read offset
>> for each stream and store them somewhere. I can probably use other
>> external
>> services (like any key/value store) to store the offset. But it would
>> introduce extra dependencies. I am thinking if I have a map of <stream,
>> offset> and periodically flush the map into a separate stream (let's say
>> __offset_ stream). With proper truncation/checkpoint mechanism, it would
>> be
>> very fast.
>> This use case here is a very standard replicated state machine. I am also
>> wondering do you guys think of providing some common library on
>> distributedlog to simply implementing state machines.
>> - KN

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message