aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Sweeney" <kevi...@apache.org>
Subject Review Request 26478: Add a flag to deduplicate storage snapshots
Date Thu, 09 Oct 2014 02:01:05 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26478/
-----------------------------------------------------------

Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji.


Bugs: AURORA-722
    https://issues.apache.org/jira/browse/AURORA-722


Repository: aurora


Description
-------

Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication
ratio on Twitter's production snapshots.

This format is backwards-incompatible, so this patch introduces a flag to control its use
(defaulting off).

This only changes the format used to write to the replicated log (where time is of the essence
since all writes are done holding the global storage lock) - the format of backups written
to disk is unchanged, as backups don't hold the lock.


Diffs
-----

  build.gradle 2e1bf78d7797f17afd51a94a22eff80e00aba464 
  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882

  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83

  src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf

  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb

  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6

  src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerFactory.java 52b2e83efc0c289366b5246aff32553a546b1a70

  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a

  src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc

  src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7

  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f

  src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/26478/diff/


Testing
-------

./gradlew -Pq build


Thanks,

Kevin Sweeney


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message