aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Sweeney" <kevi...@apache.org>
Subject Re: Review Request 26478: Add a flag to deduplicate storage snapshots
Date Wed, 15 Oct 2014 19:16:38 GMT


> On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java,
line 56
> > <https://reviews.apache.org/r/26478/diff/3/?file=721240#file721240line56>
> >
> >     'reduplicate' doesn't sit well with me.  Perhaps 'normalize' and 'denormalize'
are more standard terms that apply?  I don't feel too strongly, so don't change it if they
seem equally good to you.

I am decidedly ambivalent about the name


> On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java,
line 87
> > <https://reviews.apache.org/r/26478/diff/3/?file=721240#file721240line87>
> >
> >     This line is not covered in tests.  Please address.
> >     
> >     However, i suggest you implement this as below, and inline.
> >     
> >         ScheduledTask partialScheduledTask = scheduledTask.deepCopy();
> >         partialScheduledTask.getAssignedTask().unsetTaskConfig();
> >         return partialScheduledTask;

Inlined.


> On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java,
line 110
> > <https://reviews.apache.org/r/26478/diff/2-3/?file=716380#file716380line110>
> >
> >     Please use a better variable name.

Inlined, so no variable to name.


> On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote:
> > docs/scheduler-storage.md, line 13
> > <https://reviews.apache.org/r/26478/diff/3/?file=721234#file721234line13>
> >
> >     > Most users will want to enable both compression and deduplication.
> >     
> >     I suggest you yank this sentence out of this section, and add to the opening
paragraph:
> >     
> >     > The scheduler has two optimizations to reduce the size of snapshots and
thus improve snapshot performance: compression and deduplication.  Most users will want to
enable both compression and deduplication.

good idea, added


- Kevin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26478/#review56762
-----------------------------------------------------------


On Oct. 14, 2014, 6:32 p.m., Kevin Sweeney wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26478/
> -----------------------------------------------------------
> 
> (Updated Oct. 14, 2014, 6:32 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji.
> 
> 
> Bugs: AURORA-722
>     https://issues.apache.org/jira/browse/AURORA-722
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication
ratio on Twitter's production snapshots.
> 
> This format is backwards-incompatible, so this patch introduces a flag to control its
use (defaulting off).
> 
> This only changes the format used to write to the replicated log (where time is of the
essence since all writes are done holding the global storage lock) - the format of backups
written to disk is unchanged, as backups don't hold the lock.
> 
> 
> Diffs
> -----
> 
>   config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 
>   docs/scheduler-storage.md PRE-CREATION 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882

>   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6

>   src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906

>   src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a

>   src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc

>   src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7

>   src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f

>   src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/26478/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew -Pq build
> 
> 
> Thanks,
> 
> Kevin Sweeney
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message