mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Morozov <igm...@gmail.com>
Subject Re: Proposal for mesos replicated log improvements
Date Thu, 03 Nov 2016 17:18:45 GMT
I don't have enough context on why leveldb was choosen as a backend for
append only logs but at this moment it is deeply integrated into mesos
library. There are tools have been built around it, leveldb choice has
influenced the log interface: (beginning, ending positions), people are
doing snapshots and keep backups that are in leveldb.
If you consider leveldb to be a technical debt at this point then tackling
this debt should be a whole different discussion spanning multiple releases
something that I was not intended to do in this diff and don't have enough
context to discuss.

Hopefully that explains it.

-Igor


On Thu, Nov 3, 2016 at 4:41 AM, Joris Van Remoortere <joris@mesosphere.io>
wrote:

> I think I'm missing why we would spend energy controlling the leveldb
> compaction.
>
> I thought leveldb was used for convenience, and its log structured merge
> algorithm was just a consequence.
>
> Why not spend energy writing an append only file as opposed to optimizing
> compaction of an algorithm we don't want?
> > On Nov 3, 2016, at 2:26 AM, Jie Yu <yujie.jay@gmail.com> wrote:
> >
> > Thanks for the design doc. Those graphs look awesome. We probably should
> > get those into this doc:
> > https://github.com/apache/mesos/blob/master/docs/
> replicated-log-internals.md
> >
> > Regarding the doc, I'd like to see some correctness argument about why
> the
> > catchup process won't demote the current leader. For instance, to which
> > position do you stop the catchup and why? The correct argument does not
> > have to be very formal. Something similar to that in the following should
> > be sufficient.
> > https://github.com/apache/mesos/blob/master/docs/
> replicated-log-internals.md#catch-up
> >
> > - Jie
> >
> >> On Wed, Nov 2, 2016 at 9:35 PM, Igor Morozov <igmorv@gmail.com> wrote:
> >>
> >> Hi mesos developers,
> >>
> >> I have been working on a technical proposal to improve availability and
> >> failover strategy for mesos replicated log library. The primary use case
> >> we'd like to improve is aurora scheduler's leader failover and expensive
> >> replicated log compaction that gets run at a new leader start.
> >>
> >> Joseph Wu from Mesosphere was shepherding the proposal:
> >>
> >> https://docs.google.com/document/d/1gSn7tOzCrG6w8vCSLtQ2ruIolZNsC
> >> 4g2mi6jBsuDrHk/edit#
> >>
> >>
> >> Please review it, any feedback is highly appreciated.
> >>
> >> -Igor Morozov
> >>
>



-- 
-Igor

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message