Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOOJoEyz-vYn21VEPo=-XTpLW4pBmo3A4cQBt9cby3noODLC0w@mail.gmail.com>
References: <CAOOJoEyz-vYn21VEPo=-XTpLW4pBmo3A4cQBt9cby3noODLC0w@mail.gmail.com>
From: Zameer Manji <zmanji@apache.org>
Date: Thu, 30 Mar 2017 10:24:05 -0700
Message-ID: <CAM+cpfc7_QXbFZdqo-C-+qxfiHxOirba46vFJ9AVBSp=e9bGRg@mail.gmail.com>
Subject: Re: Future of storage in Aurora
To: dev@aurora.apache.org
Content-Type: multipart/alternative; boundary=94eb2c186a96fb910f054bf5f68c
archived-at: Thu, 30 Mar 2017 17:24:30 -0000

--94eb2c186a96fb910f054bf5f68c
Content-Type: text/plain; charset=UTF-8

I don't object to changes to storage so long as we have a migration plan
and a design doc. I'm also not opposed to radical revisits of storage,
including overhauling what we store and where we store it. For example,
instead of storing our `TaskConfig` objects could we store Mesos `TaskInfo`
objects instead? Could we store data outside of the scheduler like in
Cassandra? Should we have a high level 'Job' store to make querying for job
level data easier?

On Thu, Mar 30, 2017 at 10:16 AM, David McLaughlin <dmclaughlin@apache.org>
wrote:

> Hi all,
>
> I'd like to start a discussion around storage in Aurora.
>
> I think one of the biggest mistakes we made in migrating our storage to H2
> was deleting the memory stores as we moved. We made a pretty big bet that
> we could eventually make H2/relational databases work. I don't think that
> bet has paid off and that we need to revisit the direction we're taking.
>
> My belief is that the current H2/MyBatis approach is untenable for large
> production clusters, at least without changing our current single-master
> architecture. At Twitter we are already having to fight to keep GC
> manageable even without DbTaskStore enabled, so I don't see a path forward
> where we could eventually enable that. So far experiments with H2 off-heap
> storage have provided marginal (if any) gains.
>
> Would anyone object to restoring the in-memory stores and creating new
> implementations for the missing ones (UpdateStore)? I'd even go further and
> propose that we consider in-memory H2 and MyBatis a failed experiment and
> we drop that storage layer completely.
>
> Cheers,
> David
>
> --
> Zameer Manji
>

--94eb2c186a96fb910f054bf5f68c--