aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: Review Request 64126: Add a key-value backed StreamManager implementation
Date Fri, 01 Dec 2017 00:14:12 GMT


> On Nov. 30, 2017, 9:29 a.m., Bill Farner wrote:
> > I have an alternative approach to pose based on learnings from this patch - using
a DBMS as a key-value document store.  This could be ~2 levels of abstraction higher and involve
implementing `Storage` rather than `StreamManager`.  Similar to `LogStorage`, we would persist
mutations as they happen, and only read to sync a replica.  Warm standby could be implemented
by incrementally fetching based on modification timestamps.
> 
> Jordan Ly wrote:
>     I am +1 to this overall approach. I think this is the natural progression for durable
storage and we get a lot of needs out of the box (namely transactions).
>     
>     Warm standby is definitely nice to have but it isn't as important with the removal
of forced daily failovers for compaction.
> 
> David McLaughlin wrote:
>     I'm +1 to higher level abstractions. I don't like the idea of having to assume an
underlying Log-based format for durability - and instead have abstractions that assume (or
at least expect) capabilities.
> 
> Stephan Erb wrote:
>     Going directly to a key-value interface sounds also nicer to me than using it behind
a log abstraction.
>     
>     One thing I am wondering though: Why do you want to go to a DBMS? While ZK is not
a real database, anything in particular that you don't like for this usecase?

> Why do you want to go to a DBMS? While ZK is not a real database, anything in particular
that you don't like for this usecase?

The short answer: transactions.  The longer answer: a first-class database would eliminate
the need for much of the logic you see in this patch (synthetic transactions, LogEntry framing,
key hashing, namespace fanout mitigation).  The backup/recovery story is also much more obvious.

With the new route proposed in this thread, i may still proceed with ZK as the stock replacement
for LogStorage but without as much effort for edge cases of huge records, high scale, and
warm failover.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64126/#review192310
-----------------------------------------------------------


On Nov. 28, 2017, 5:40 p.m., Bill Farner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64126/
> -----------------------------------------------------------
> 
> (Updated Nov. 28, 2017, 5:40 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This introduces an alternative implementation of `StreamManager`, which is the
> primary persistence mechanism used in `LogStorage`.  There are two distinct parts
> of this patch:
> 
>   - a storage 'business logic' layer that exposes a stream-oriented append-only
>     storage interface.  This contains components to:
>     - extract keys for any log `Op`
>     - hash keys to constrain the keyspace
>     - [de]serialize `LogEntries` and limit record sizes (using existing Framing)
>     - provide simple transactional behavior
>   - a key-value store interface, with an implementation based on ZooKeeper
> 
> This achieves the following results:
> 
>   - no inherent need for periodic Snapshot or failover to cull storage
>   - atomic multi-operation transactions even for a key-value store that does not
>     support multi-key transactions
>   - storage operations that amortize to a single storage round-trip
>     (in the case of ZooKeeper, using ZK `multi()`)
>   - straightforward path to warm standby for failover
> 
> 
> **Note**: there are still a few rough edges that must be rounded out before this can
be used even in integration tests, but the storage 'business logic' layers are reasonably
polished and well-tested.  I opted to use this as a moment to pause and gather feedback.
> 
> **Reviewers**: I suggest starting with `AbstractStreamManagerTest.java` as it most succinctly
captures the features sought in the new layers.  A first pass review on the general approach
would be most useful, after which we can move on to specifics.
> 
> 
> Diffs
> -----
> 
>   build.gradle af119910e84c48f75f2573ababcfa287c3b986fc 
>   commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java
a4504b8d1307ca15de3bb92674b8d77c6aea8467 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 3c12532acca9107f5a54df48fc59348d70af29c6

>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 7ffcf4f471b97e32426c82972472115c5c5c4d02

>   src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68

>   src/main/java/org/apache/aurora/scheduler/storage/log/AbstractStreamTransaction.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/Entries.java ab7cd2d40013827122e27275f557e9c8fff9dbb7

>   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 90c5383febc0ac49b0bf8b369fe0107f1fa8a6b3

>   src/main/java/org/apache/aurora/scheduler/storage/log/Framing.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java c8dc7ad141f27318f18afdc440250afc42f0bc6d

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java ea147c0ba6aaa6d113144be0a8330bd2f73d2453

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerFactory.java 52b2e83efc0c289366b5246aff32553a546b1a70

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java baf2647c54f1f9918139584e5151873a3853e83c

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerModule.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamTransaction.java a51fd18ad00537bf244419442078548d8545a841

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/AppendOnlyStore.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/HashedKeyValueStore.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStore.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStreamModule.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyedStreamManager.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogEntrySerializingStore.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogOpStore.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/TransactionalKeyValueStore.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyValueStore.java
PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4

>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 5cb5310ed096ca1fb47b980401e3712948271ac4

>   src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 3f445595a81a5655c6c486791a9b55d8dc7f2f76

>   src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java f43a836a4176c465fa3f496c77ef6dc1cbc1691a

>   src/test/java/org/apache/aurora/scheduler/storage/log/kv/AbstractStreamManagerTest.java
PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/storage/log/kv/InMemoryStreamManagerTest.java
PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueNonVolatileStorageTest.java
PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/storage/log/kv/MapKeyValueStore.java PRE-CREATION

>   src/test/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyedStreamManagerTest.java
PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/64126/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Bill Farner
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message