aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Review Request 64126: Add a key-value backed StreamManager implementation
Date Wed, 29 Nov 2017 01:40:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64126/
-----------------------------------------------------------

Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.


Repository: aurora


Description
-------

This introduces an alternative implementation of `StreamManager`, which is the
primary persistence mechanism used in `LogStorage`.  There are two distinct parts
of this patch:

  - a storage 'business logic' layer that exposes a stream-oriented append-only
    storage interface.  This contains components to:
    - extract keys for any log `Op`
    - hash keys to constrain the keyspace
    - [de]serialize `LogEntries` and limit record sizes (using existing Framing)
    - provide simple transactional behavior
  - a key-value store interface, with an implementation based on ZooKeeper

This achieves the following results:

  - no inherent need for periodic Snapshot or failover to cull storage
  - atomic multi-operation transactions even for a key-value store that does not
    support multi-key transactions
  - storage operations that amortize to a single storage round-trip
    (in the case of ZooKeeper, using ZK `multi()`)
  - straightforward path to warm standby for failover


**Note**: there are still a few rough edges that must be rounded out before this can be used
even in integration tests, but the storage 'business logic' layers are reasonably polished
and well-tested.  I opted to use this as a moment to pause and gather feedback.

**Reviewers**: I suggest starting with `AbstractStreamManagerTest.java` as it most succinctly
captures the features sought in the new layers.  A first pass review on the general approach
would be most useful, after which we can move on to specifics.


Diffs
-----

  build.gradle af119910e84c48f75f2573ababcfa287c3b986fc 
  commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java
a4504b8d1307ca15de3bb92674b8d77c6aea8467 
  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 3c12532acca9107f5a54df48fc59348d70af29c6

  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 7ffcf4f471b97e32426c82972472115c5c5c4d02

  src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68

  src/main/java/org/apache/aurora/scheduler/storage/log/AbstractStreamTransaction.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/Entries.java ab7cd2d40013827122e27275f557e9c8fff9dbb7

  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 90c5383febc0ac49b0bf8b369fe0107f1fa8a6b3

  src/main/java/org/apache/aurora/scheduler/storage/log/Framing.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java c8dc7ad141f27318f18afdc440250afc42f0bc6d

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java ea147c0ba6aaa6d113144be0a8330bd2f73d2453

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerFactory.java 52b2e83efc0c289366b5246aff32553a546b1a70

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java baf2647c54f1f9918139584e5151873a3853e83c

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerModule.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamTransaction.java a51fd18ad00537bf244419442078548d8545a841

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/AppendOnlyStore.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/HashedKeyValueStore.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStore.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueStreamModule.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/KeyedStreamManager.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogEntrySerializingStore.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/kv/LogOpStore.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/kv/TransactionalKeyValueStore.java
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyValueStore.java
PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4

  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 5cb5310ed096ca1fb47b980401e3712948271ac4

  src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 3f445595a81a5655c6c486791a9b55d8dc7f2f76

  src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java f43a836a4176c465fa3f496c77ef6dc1cbc1691a

  src/test/java/org/apache/aurora/scheduler/storage/log/kv/AbstractStreamManagerTest.java
PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/storage/log/kv/InMemoryStreamManagerTest.java
PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/storage/log/kv/KeyValueNonVolatileStorageTest.java
PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/storage/log/kv/MapKeyValueStore.java PRE-CREATION

  src/test/java/org/apache/aurora/scheduler/storage/log/kv/zookeeper/ZooKeeperKeyedStreamManagerTest.java
PRE-CREATION 


Diff: https://reviews.apache.org/r/64126/diff/1/


Testing
-------


Thanks,

Bill Farner


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message