distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <sij...@twitter.com.INVALID>
Subject Re: Use DL stream to store offsets?
Date Wed, 02 Nov 2016 17:38:36 GMT
On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnasimm@gmail.com>

> As part of implementing the kafka subscriber interface, I am wondering is
> there anyone use DL stream for storing the offsets?
> For example, if I have N streams (0..N-1), I need to track the read offset
> for each stream and store them somewhere. I can probably use other external
> services (like any key/value store) to store the offset. But it would
> introduce extra dependencies. I am thinking if I have a map of <stream,
> offset> and periodically flush the map into a separate stream (let's say
> __offset_ stream). With proper truncation/checkpoint mechanism, it would be
> very fast.
> This use case here is a very standard replicated state machine. I am also
> wondering do you guys think of providing some common library on
> distributedlog to simply implementing state machines.

Yes, one of the goals for this project is to make building state machines
easier using DL. As what Leigh said, we are working on some common
libraries for this.

> - KN

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message