distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khurrum Nasim <khurrumnas...@gmail.com>
Subject Re: Use DL stream to store offsets?
Date Thu, 03 Nov 2016 07:49:21 GMT
That's cool, Leigh. It would be good that you can have some basic version
out as soon as possible. I'd like to leverage any existing solution, rather
than building a separate one.

- KN

On Wed, Nov 2, 2016 at 9:19 AM, Leigh Stewart <lstewart@twitter.com.invalid>
wrote:

> Whats your timeline Khurrum? Maybe we can work something out.
>
> On Wed, Nov 2, 2016 at 8:15 AM, Leigh Stewart <lstewart@twitter.com>
> wrote:
>
> > We have in fact built something like this. No plans as yet to release,
> but
> > I think we would like to eventually.
> >
> > On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnasimm@gmail.com>
> > wrote:
> >
> >> As part of implementing the kafka subscriber interface, I am wondering
> is
> >> there anyone use DL stream for storing the offsets?
> >>
> >> For example, if I have N streams (0..N-1), I need to track the read
> offset
> >> for each stream and store them somewhere. I can probably use other
> >> external
> >> services (like any key/value store) to store the offset. But it would
> >> introduce extra dependencies. I am thinking if I have a map of <stream,
> >> offset> and periodically flush the map into a separate stream (let's say
> >> __offset_ stream). With proper truncation/checkpoint mechanism, it would
> >> be
> >> very fast.
> >>
> >> This use case here is a very standard replicated state machine. I am
> also
> >> wondering do you guys think of providing some common library on
> >> distributedlog to simply implementing state machines.
> >>
> >> - KN
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message