aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reza Motamedi <reza.motam...@gmail.com>
Subject Re: Review Request 55217: AURORA-1847 Eliminate sequential scan in MemTaskStore.getJobKeys()
Date Sun, 08 Jan 2017 08:43:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55217/#review160830
-----------------------------------------------------------


Ship it!




Ship It!

- Reza Motamedi


On Jan. 5, 2017, 6:59 p.m., Mehrdad Nurolahzade wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55217/
> -----------------------------------------------------------
> 
> (Updated Jan. 5, 2017, 6:59 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1847
>     https://issues.apache.org/jira/browse/AURORA-1847
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> If scheduler is configured to run with the `MemTaskStore` every hit on scheduler landing
page (`/scheduler`) causes a call to `MemTaskStore.getJobKeys()` through `ReadOnlyScheduler.getRoleSummary()`.
> 
> The implementation of `MemTaskStore.getJobKeys()` is currently very inefficient as it
requires a sequential scan of the task store and mapping to their respective job keys. In
Twitter clusters this method is currently taking half a second per call (`mem_storage_get_job_keys`).

> 
> This patch eliminates the sequential scan and mapping to job key by simply returning
an immutable copy of the key set of the existing secondary index `job`.
> 
> 
> Diffs
> -----
> 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java f2f00b92bf901c438e40bbc177e9f5a112be1bbc

>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java fc272ddb45be8b2f55f01c3d54f7fa9058202c0b

> 
> Diff: https://reviews.apache.org/r/55217/diff/
> 
> 
> Testing
> -------
> 
> Using the following modified version of existing `TaskStoreBenchmarks.MemFetchTasksBenchmark`
which benchmarks `TaskStore.getJobKeys()`:
> ```java
> public static class MemFetchTasksBenchmark extends AbstractFetchTasksBenchmark {
>     @Setup(Level.Trial)
>     @Override
>     public void setUp() {
>       storage = Guice.createInjector(
>           Modules.combine(
>               DbModule.testModuleWithWorkQueue(PLAIN, Optional.of(new InMemStoresModule(PLAIN))),
>               new AbstractModule() {
>                 @Override
>                 protected void configure() {
>                   bind(StatsProvider.class).toInstance(new FakeStatsProvider());
>                   bind(Clock.class).toInstance(new FakeClock());
>                 }
>               }))
>           .getInstance(Storage.class);
> 
>     }
> 
>     @Setup(Level.Iteration)
>     public void setUpIteration() {
>       createTasks(numTasks);
>     }
> 
>     @TearDown(Level.Iteration)
>     public void tearDownIteration() {
>       deleteTasks();
>     }
> 
>     @Benchmark
>     public Iterable<IJobKey> run() {
>       return storage.read(store -> store.getTaskStore().getJobKeys());
>     }
>   }
> ```
> 
> Benchmark results BEFORE patch:
> ```
> Benchmark                                       (numTasks)   Mode  Cnt    Score     Error
 Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       10000  thrpt    5  572.761 ± 168.865
 ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       50000  thrpt    5   80.697 ±  
8.516  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run      100000  thrpt    5   25.102 ±  
3.244  ops/s
> ```
> 
> Benchmark results AFTER patch:
> ```
> Benchmark                                       (numTasks)   Mode  Cnt       Score  
    Error  Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       10000  thrpt    5  327336.998 ±
10207.402  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run       50000  thrpt    5  320506.958 ±
23430.527  ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run      100000  thrpt    5  287962.695 ±
51917.245  ops/s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message