aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zameer Manji" <zma...@apache.org>
Subject Re: Review Request 42041: Enable H2 query statistics collection.
Date Fri, 08 Jan 2016 21:52:58 GMT


> On Jan. 7, 2016, 3:21 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java, line 126
> > <https://reviews.apache.org/r/42041/diff/1/?file=1186513#file1186513line126>
> >
> >     Are there any perf implications from having it ON by default? Should it be a
configurable flag instead?
> 
> Zameer Manji wrote:
>     Good question, I'll run benchmarks and update the testing done.
> 
> John Sirois wrote:
>     And if there are perf implications, there is always [`SET QUERY_STATISTICS`](http://www.h2database.com/html/grammar.html?highlight=QUERY_STATISTICS&search=QUERY_STATISTICS#set_query_statistics)
via the /h2console - and maybe that could just be documented as an ad-hoc option.
> 
> Zameer Manji wrote:
>     John, using `SET QUERY_STATISTICS` will not persist across failovers and is not retroactive.
This means operators need to run `SET QUERY_STATISTICS` and wait for data to be collected
if they notice the scheduler is slower than expected. If this option is set when the database
is created the data collected should reflect all operations done by the scheduler and that
is more useful for triaging slow scheduling in large clusters.
>     
>     I think providing a flag or enabling it by default (pending results of benchmarks)
are better solutions.
> 
> John Sirois wrote:
>     Makes sense.  Have you thought about the existing slow query logging and how that
evolves with this?  Probably fine side-by-side, but maybe confusing and less valuable once
the mem store is gone and the db store is all we have.

I have updated the testing done with benchmarks and the change appears to be small for scheduling.
I think the utility of query statistics justifies turning it on by default.

John, I think slow query logging can be removed once the mem store has been removed.


- Zameer


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42041/#review113355
-----------------------------------------------------------


On Jan. 8, 2016, 1:52 p.m., Zameer Manji wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42041/
> -----------------------------------------------------------
> 
> (Updated Jan. 8, 2016, 1:52 p.m.)
> 
> 
> Review request for Aurora, John Sirois and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> With this enabled operators can visit the H2 console at /h2console and run queries like
`SELECT * FROM INFORMATION_SCHEMA.QUERY_STATISTICS ORDER BY MAX_EXECUTION_TIME DESC;` to diagnose
slow schedulers.
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java e3efbdb80d8b18312bf1589eef883cdeee65b225

> 
> Diff: https://reviews.apache.org/r/42041/diff/
> 
> 
> Testing
> -------
> 
> Ran `SELECT * FROM INFORMATION_SCHEMA.QUERY_STATISTICS ORDER BY MAX_EXECUTION_TIME DESC;`
within vagrant and saw query statistics.
> 
> Benchmarks
> 
> Master (c595228):
> Benchmark                                                                     (numPendingTasks)
  Mode  Cnt      Score      Error  Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark                   
       N/A  thrpt   10  64138.084 ± 6732.130  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark          
       N/A  thrpt   10  23863.861 ± 2101.622  ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark        
       N/A  thrpt   10   2228.883 ±  311.434  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
         1  thrpt   10     50.914 ±    2.488  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
        10  thrpt   10     43.729 ±    3.038  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
       100  thrpt   10     44.409 ±    4.426  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
      1000  thrpt   10     40.429 ±    7.526  ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark        
       N/A  thrpt   10  22942.538 ± 1281.331  ops/s
> 
> 
> This change:
> Benchmark                                                                     (numPendingTasks)
  Mode  Cnt      Score      Error  Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark                   
       N/A  thrpt   10  65285.628 ± 2422.816  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark          
       N/A  thrpt   10  24573.332 ± 1332.474  ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark        
       N/A  thrpt   10   2430.402 ±  258.860  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
         1  thrpt   10     43.810 ±    2.669  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
        10  thrpt   10     37.378 ±   14.637  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
       100  thrpt   10     40.180 ±    9.738  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                      
      1000  thrpt   10     24.130 ±   15.746  ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark        
       N/A  thrpt   10  18429.830 ± 3077.426  ops/s
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message