distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khurrum Nasim <khurrumnas...@gmail.com>
Subject Use DL stream to store offsets?
Date Wed, 02 Nov 2016 09:43:35 GMT
As part of implementing the kafka subscriber interface, I am wondering is
there anyone use DL stream for storing the offsets?

For example, if I have N streams (0..N-1), I need to track the read offset
for each stream and store them somewhere. I can probably use other external
services (like any key/value store) to store the offset. But it would
introduce extra dependencies. I am thinking if I have a map of <stream,
offset> and periodically flush the map into a separate stream (let's say
__offset_ stream). With proper truncation/checkpoint mechanism, it would be
very fast.

This use case here is a very standard replicated state machine. I am also
wondering do you guys think of providing some common library on
distributedlog to simply implementing state machines.

- KN

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message