distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leigh Stewart <lstew...@twitter.com.INVALID>
Subject Re: Use DL stream to store offsets?
Date Fri, 18 Nov 2016 15:08:33 GMT
Maybe sometime next quarter-- we're still in the process of standing up the
first production service to use this library.
No reason not to work on another implementation though. The requirements
will probably be a little different.
We'd be happy to help you understand how to approach error handling and
consistency if you do give it a shot.

On Wed, Nov 9, 2016 at 2:38 AM, Jay Juma <jayk.juma@gmail.com> wrote:

> I am also interested in a framework/library that makes building state
> machine easier. When do you think you can share that? I'd like to
> contribute too.
>
> - Jay
>
> On Wed, Nov 2, 2016 at 9:19 AM, Leigh Stewart <lstewart@twitter.com.invalid
> >
> wrote:
>
> > Whats your timeline Khurrum? Maybe we can work something out.
> >
> > On Wed, Nov 2, 2016 at 8:15 AM, Leigh Stewart <lstewart@twitter.com>
> > wrote:
> >
> > > We have in fact built something like this. No plans as yet to release,
> > but
> > > I think we would like to eventually.
> > >
> > > On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnasimm@gmail.com
> >
> > > wrote:
> > >
> > >> As part of implementing the kafka subscriber interface, I am wondering
> > is
> > >> there anyone use DL stream for storing the offsets?
> > >>
> > >> For example, if I have N streams (0..N-1), I need to track the read
> > offset
> > >> for each stream and store them somewhere. I can probably use other
> > >> external
> > >> services (like any key/value store) to store the offset. But it would
> > >> introduce extra dependencies. I am thinking if I have a map of
> <stream,
> > >> offset> and periodically flush the map into a separate stream (let's
> say
> > >> __offset_ stream). With proper truncation/checkpoint mechanism, it
> would
> > >> be
> > >> very fast.
> > >>
> > >> This use case here is a very standard replicated state machine. I am
> > also
> > >> wondering do you guys think of providing some common library on
> > >> distributedlog to simply implementing state machines.
> > >>
> > >> - KN
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message