aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zameer Manji <zma...@apache.org>
Subject Re: Review Request 44471: Implementing dbsnapshotting
Date Tue, 08 Mar 2016 16:49:43 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44471/#review122541
-----------------------------------------------------------


Ship it!




I have one objection to the future of this patch which is dropping the other fields in the
snapshot after we move all in memory storage to H2.

I think keeping those fields for a while will be useful because they allow cluster operators
to look at a snapshot of the cluster state and perform analytics, etc.

- Zameer Manji


On March 7, 2016, 3:55 p.m., Maxim Khutornenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44471/
> -----------------------------------------------------------
> 
> (Updated March 7, 2016, 3:55 p.m.)
> 
> 
> Review request for Aurora, Bill Farner and Zameer Manji.
> 
> 
> Bugs: AURORA-1627
>     https://issues.apache.org/jira/browse/AURORA-1627
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> H2 DB is now `script`ed into a dedicated `dbScript` field in `Snapshot` thrift struct
to speed up recovery under H2 MVStore.
> 
> There was extra care needed to ensure task stores are functioning properly when they
are toggled between in-memory and DB mode. Tried to document as much as possible. Let me know
if something is still unclear. The existing thrift snapshot fields will be supported until
all stores are fully migrated to H2 and removed subsequently along with in-memory store code.
> 
> `SnapshotStoreImplTest` got rewritten into `SnapshotStoreImplIT` to fully test `TaskStore`
migration matrix.
> 
> Also, moved checkstyle suppressions into annotations for more precise control.
> 
> Master:
> ```
> Benchmark                                                   (updateCount)   Mode  Cnt
 Score   Error  Units
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run              1  thrpt    5
 4.876 ± 0.403  ops/s
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run              5  thrpt    5
 0.942 ± 0.032  ops/s
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run             10  thrpt    5
 0.436 ± 0.040  ops/s
> ```
> 
> This patch:
> ```
> Benchmark                                                   (updateCount)   Mode  Cnt
 Score   Error  Units
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run              1  thrpt    5
 9.846 ± 0.886  ops/s
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run              5  thrpt    5
 1.977 ± 0.319  ops/s
> SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run             10  thrpt    5
 1.310 ± 0.139  ops/s
> ```
> 
> 
> Diffs
> -----
> 
>   api/src/main/thrift/org/apache/aurora/gen/storage.thrift 6dc46147bb0703e83a210a81ee24081183389a89

>   config/checkstyle/checkstyle.xml 2074beb66ca19eb2da3d444bc64e583409e30ead 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 50044e1c38c712a52855dd96fa2a9de769d6c973

>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java ca484fab2a9b136c6d5b9be31e1ad1a5360f1b7a

>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 92849d9a82dcd6d5fe6cc452061bcdca31415f40

>   src/main/java/org/apache/aurora/scheduler/storage/Storage.java 5124d17c9bc755e33a3e97b914caed34b77cdb94

>   src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 46b3d104931f175314902ccd39b81c4b6d67d4f3

>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 6d8fa11d0aed841388e1e87fc0acdaf6b8f42d06

>   src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java cca92dd6e9e52f9e38394c4c688ab4897eb8e23f

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java ed63a7471d654dcefd2ff24e2e462974883541f2

>   src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java db90150b48c5b134dde6c69f70ebca82bfdd0c12

>   src/main/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorage.java 2f07afb73f6e585a6b43be68134e2beecac83d31

>   src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 172dd206d09f131807cb33fe841ca6ebc8198a14

>   src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java PRE-CREATION

>   src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplTest.java 44078675286dec908da7a404ae8fce7c495a7019

>   src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 75130a315573dbb9eff1a9b1c5408df27623cf54

> 
> Diff: https://reviews.apache.org/r/44471/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew -Pq build
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message