distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <sij...@twitter.com.INVALID>
Subject Re: Use DL stream to store offsets?
Date Wed, 02 Nov 2016 17:38:36 GMT
On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnasimm@gmail.com>
wrote:

> As part of implementing the kafka subscriber interface, I am wondering is
> there anyone use DL stream for storing the offsets?
>
> For example, if I have N streams (0..N-1), I need to track the read offset
> for each stream and store them somewhere. I can probably use other external
> services (like any key/value store) to store the offset. But it would
> introduce extra dependencies. I am thinking if I have a map of <stream,
> offset> and periodically flush the map into a separate stream (let's say
> __offset_ stream). With proper truncation/checkpoint mechanism, it would be
> very fast.
>
> This use case here is a very standard replicated state machine. I am also
> wondering do you guys think of providing some common library on
> distributedlog to simply implementing state machines.
>

Yes, one of the goals for this project is to make building state machines
easier using DL. As what Leigh said, we are working on some common
libraries for this.


>
> - KN
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message