Return-Path: X-Original-To: apmail-aurora-reviews-archive@minotaur.apache.org Delivered-To: apmail-aurora-reviews-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4D658181DF for ; Tue, 8 Mar 2016 16:49:45 +0000 (UTC) Received: (qmail 12461 invoked by uid 500); 8 Mar 2016 16:49:45 -0000 Delivered-To: apmail-aurora-reviews-archive@aurora.apache.org Received: (qmail 12410 invoked by uid 500); 8 Mar 2016 16:49:45 -0000 Mailing-List: contact reviews-help@aurora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@aurora.apache.org Delivered-To: mailing list reviews@aurora.apache.org Received: (qmail 12380 invoked by uid 99); 8 Mar 2016 16:49:44 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2016 16:49:44 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id D49FF2AA87C; Tue, 8 Mar 2016 16:49:43 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7528273920101120681==" MIME-Version: 1.0 Subject: Re: Review Request 44471: Implementing dbsnapshotting From: Zameer Manji To: Bill Farner , Zameer Manji Cc: Aurora , Maxim Khutornenko Date: Tue, 08 Mar 2016 16:49:43 -0000 Message-ID: <20160308164943.8638.56066@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Zameer Manji X-ReviewGroup: Aurora X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/44471/ X-Sender: Zameer Manji X-ReviewBoard-ShipIt: 1 References: <20160307235559.8639.39351@reviews.apache.org> In-Reply-To: <20160307235559.8639.39351@reviews.apache.org> X-ReviewBoard-Diff-For: src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java X-ReviewBoard-Diff-For: src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplTest.java Reply-To: Zameer Manji X-ReviewRequest-Repository: aurora --===============7528273920101120681== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44471/#review122541 ----------------------------------------------------------- Ship it! I have one objection to the future of this patch which is dropping the other fields in the snapshot after we move all in memory storage to H2. I think keeping those fields for a while will be useful because they allow cluster operators to look at a snapshot of the cluster state and perform analytics, etc. - Zameer Manji On March 7, 2016, 3:55 p.m., Maxim Khutornenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44471/ > ----------------------------------------------------------- > > (Updated March 7, 2016, 3:55 p.m.) > > > Review request for Aurora, Bill Farner and Zameer Manji. > > > Bugs: AURORA-1627 > https://issues.apache.org/jira/browse/AURORA-1627 > > > Repository: aurora > > > Description > ------- > > H2 DB is now `script`ed into a dedicated `dbScript` field in `Snapshot` thrift struct to speed up recovery under H2 MVStore. > > There was extra care needed to ensure task stores are functioning properly when they are toggled between in-memory and DB mode. Tried to document as much as possible. Let me know if something is still unclear. The existing thrift snapshot fields will be supported until all stores are fully migrated to H2 and removed subsequently along with in-memory store code. > > `SnapshotStoreImplTest` got rewritten into `SnapshotStoreImplIT` to fully test `TaskStore` migration matrix. > > Also, moved checkstyle suppressions into annotations for more precise control. > > Master: > ``` > Benchmark (updateCount) Mode Cnt Score Error Units > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 1 thrpt 5 4.876 ± 0.403 ops/s > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 5 thrpt 5 0.942 ± 0.032 ops/s > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 10 thrpt 5 0.436 ± 0.040 ops/s > ``` > > This patch: > ``` > Benchmark (updateCount) Mode Cnt Score Error Units > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 1 thrpt 5 9.846 ± 0.886 ops/s > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 5 thrpt 5 1.977 ± 0.319 ops/s > SnapshotBenchmarks.RestoreSnapshotWithUpdatesBenchmark.run 10 thrpt 5 1.310 ± 0.139 ops/s > ``` > > > Diffs > ----- > > api/src/main/thrift/org/apache/aurora/gen/storage.thrift 6dc46147bb0703e83a210a81ee24081183389a89 > config/checkstyle/checkstyle.xml 2074beb66ca19eb2da3d444bc64e583409e30ead > src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 50044e1c38c712a52855dd96fa2a9de769d6c973 > src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java ca484fab2a9b136c6d5b9be31e1ad1a5360f1b7a > src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 92849d9a82dcd6d5fe6cc452061bcdca31415f40 > src/main/java/org/apache/aurora/scheduler/storage/Storage.java 5124d17c9bc755e33a3e97b914caed34b77cdb94 > src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 46b3d104931f175314902ccd39b81c4b6d67d4f3 > src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 6d8fa11d0aed841388e1e87fc0acdaf6b8f42d06 > src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java cca92dd6e9e52f9e38394c4c688ab4897eb8e23f > src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java ed63a7471d654dcefd2ff24e2e462974883541f2 > src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java db90150b48c5b134dde6c69f70ebca82bfdd0c12 > src/main/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorage.java 2f07afb73f6e585a6b43be68134e2beecac83d31 > src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 172dd206d09f131807cb33fe841ca6ebc8198a14 > src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java PRE-CREATION > src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplTest.java 44078675286dec908da7a404ae8fce7c495a7019 > src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 75130a315573dbb9eff1a9b1c5408df27623cf54 > > Diff: https://reviews.apache.org/r/44471/diff/ > > > Testing > ------- > > ./gradlew -Pq build > ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh > > > Thanks, > > Maxim Khutornenko > > --===============7528273920101120681==--