aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sirois <jsir...@apache.org>
Subject Re: Review Request 43457: Increase throughput of DbTaskStore
Date Fri, 12 Feb 2016 15:37:44 GMT


> On Feb. 10, 2016, 5:23 p.m., John Sirois wrote:
> > > and removes the population of an object via a constructor which is slower than
populating an object via setters.
> > 
> > For clarification, is this optimization the much smaller of the 2?  Naively - I'd
hope construction would beat setters, but - if not - be _very close_.  If this latter bit
was significant, my faith in the jvm is duly shaken.
> 
> Zameer Manji wrote:
>     Switching to Pair and populating via constructor:
>     ````
>     Benchmark                                      (numTasks)   Mode  Cnt   Score   
Error  Units
>     TaskStoreBenchmarks.DBFetchTasksBenchmark.run       10000  thrpt    5  53.040 ±
17.152  ops/s
>     TaskStoreBenchmarks.DBFetchTasksBenchmark.run       50000  thrpt    5   3.939 ±
 3.357  ops/s
>     TaskStoreBenchmarks.DBFetchTasksBenchmark.run      100000  thrpt    5   0.079 ±
 0.020  ops/s
>     ````
>     
>     I suspect it is because if you populate via constructor you cannot use `<id>`.
> 
> John Sirois wrote:
>     Fwict only the 100000 case has a statistically significant win (the error bars in
the 2nd run are quite massive) - is that worth it?  Does Aurora actually fetch 100K tasks
for any operation in practice at Twitter?
> 
> Zameer Manji wrote:
>     Yes this change is worth it. At Twitter Aurora actually fetches at least 100K tasks
for some operations. This is a side effect of some unscoped `fetchTask` queries we currently
have.
> 
> John Sirois wrote:
>     I won't block, but you can still use id - see constructor/idArg: http://www.mybatis.org/mybatis-3/sqlmap-xml.html
>     It would be nice to either use Pairs or not use them and this will leave is with
one use for task links.

OK - you apparently understood this already - it looks like id columns are only used internally
by mybatis for property-style but they are expected to be a real constructor argument for
constructor-style so constructor idArg is a no-go for Pair.


- John


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43457/#review118772
-----------------------------------------------------------


On Feb. 11, 2016, 1:03 p.m., Zameer Manji wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43457/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 1:03 p.m.)
> 
> 
> Review request for Aurora, John Sirois and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Profiling master indicated that the bottleneck was MyBatis populating ResultSets and
populating the resulting objects. This patch removes subselects, which reduces the number
of ResultSets and removes the population of an object via a constructor which is slower than
populating an object via setters.
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbAssginedPort.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbAssignedTask.java 93722395ed9fcd22dcb12e34e648e6e410952d43

>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbScheduledTask.java 502a1fa6fc141df498f0f09af292ce24e269731d

>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml b1394cf44b7ddafcbc47bb1968306d0b33293380

>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml ea469cce31544221c34ae05a1c65f71271985655

> 
> Diff: https://reviews.apache.org/r/43457/diff/
> 
> 
> Testing
> -------
> 
> Master:
> Benchmark                                      (numTasks)   Mode  Cnt   Score    Error
 Units
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run       10000  thrpt    5  44.052 ± 14.689
 ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run       50000  thrpt    5   0.179 ±  0.052
 ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run      100000  thrpt    5   0.087 ±  0.022
 ops/s
> 
> This Patch:
> Benchmark                                      (numTasks)   Mode  Cnt   Score   Error
 Units
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run       10000  thrpt    5  51.531 ± 7.236
 ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run       50000  thrpt    5   7.370 ± 1.320
 ops/s
> TaskStoreBenchmarks.DBFetchTasksBenchmark.run      100000  thrpt    5   2.143 ± 1.234
 ops/s
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message