Mailing-List: contact reviews-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: reviews@aurora.apache.org
Content-Type: multipart/alternative;
 boundary="===============8522825554702413619=="
MIME-Version: 1.0
Subject: Re: Review Request 55217: AURORA-1847 Eliminate sequential scan in
 MemTaskStore.getJobKeys()
From: Reza Motamedi <reza.motamedi@gmail.com>
To: Joshua Cohen <jcohen@apache.org>, Stephan Erb <serb@apache.org>
Cc: Aurora <reviews@aurora.apache.org>,
 Mehrdad Nurolahzade <mehrdad@nurolahzade.com>,
 Reza Motamedi <reza.motamedi@gmail.com>
Date: Sun, 08 Jan 2017 08:43:03 -0000
Message-ID: <20170108084303.20794.92386@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Reza Motamedi <noreply@reviews.apache.org>
References: <20170105185909.13569.73799@reviews.apache.org>
In-Reply-To: <20170105185909.13569.73799@reviews.apache.org>
Reply-To: Reza Motamedi <reza.motamedi@gmail.com>
archived-at: Sun, 08 Jan 2017 08:43:07 -0000

--===============8522825554702413619==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55217/#review160830
-----------------------------------------------------------


Ship it!


Ship It!

- Reza Motamedi


On Jan. 5, 2017, 6:59 p.m., Mehrdad Nurolahzade wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55217/
> -----------------------------------------------------------
> 
> (Updated Jan. 5, 2017, 6:59 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1847
>     https://issues.apache.org/jira/browse/AURORA-1847
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> If scheduler is configured to run with the `MemTaskStore` every hit on scheduler landing page (`/scheduler`) causes a call to `MemTaskStore.getJobKeys()` through `ReadOnlyScheduler.getRoleSummary()`.
> 
> The implementation of `MemTaskStore.getJobKeys()` is currently very inefficient as it requires a sequential scan of the task store and mapping to their respective job keys. In Twitter clusters this method is currently taking half a second per call (`mem_storage_get_job_keys`). 
> 
> This patch eliminates the sequential scan and mapping to job key by simply returning an immutable copy of the key set of the existing secondary index `job`.
> 
> 
> Diffs
> -----
> 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java f2f00b92bf901c438e40bbc177e9f5a112be1bbc 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java fc272ddb45be8b2f55f01c3d54f7fa9058202c0b 
> 
> Diff: https://reviews.apache.org/r/55217/diff/
> 
> 
> Testing
> -------
> 
> Using the following modified version of existing `TaskStoreBenchmarks.MemFetchTasksBenchmark` which benchmarks `TaskStore.getJobKeys()`:
> ```java
> public static class MemFetchTasksBenchmark extends AbstractFetchTasksBenchmark {
>     @Setup(Level.Trial)
>     @Override
>     public void setUp() {
>       storage = Guice.createInjector(
>           Modules.combine(
>               DbModule.testModuleWithWorkQueue(PLAIN, Optional.of(new InMemStoresModule(PLAIN))),
>               new AbstractModule() {
>                 @Override
>                 protected void configure() {
>                   bind(StatsProvider.class).toInstance(new FakeStatsProvider());
>                   bind(Clock.class).toInstance(new FakeClock());
>                 }
>               }))
>           .getInstance(Storage.class);
> 
>     }
> 
>     @Setup(Level.Iteration)
>     public void setUpIteration() {
>       createTasks(numTasks);
>     }
> 
>     @TearDown(Level.Iteration)
>     public void tearDownIteration() {
>       deleteTasks();
>     }
> 
>     @Benchmark
>     public Iterable<IJobKey> run() {
>       return storage.read(store -> store.getTaskStore().getJobKeys());
>     }
>   }
> ```
> 
> Benchmark results BEFORE patch:
> ```
> Benchmark                                       (numTasks)   Mode  Cnt    Score     Error  Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       10000  thrpt    5  572.761 ± 168.865  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       50000  thrpt    5   80.697 ±   8.516  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run      100000  thrpt    5   25.102 ±   3.244  ops/s
> ```
> 
> Benchmark results AFTER patch:
> ```
> Benchmark                                       (numTasks)   Mode  Cnt       Score       Error  Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       10000  thrpt    5  327336.998 ± 10207.402  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       50000  thrpt    5  320506.958 ± 23430.527  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run      100000  thrpt    5  287962.695 ± 51917.245  ops/s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>


--===============8522825554702413619==--