aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: Review Request 64288: Add a SQL persistence implementation
Date Mon, 04 Dec 2017 19:55:36 GMT


> On Dec. 3, 2017, 6:29 p.m., David McLaughlin wrote:
> > Do we have any idea how this performs at scale? I'd like to avoid repeating the
MyBatis work, where we delayed the scale testing until after we had committed a bunch of patches.

A fair concern!  Until i can perform a more real-world test, here's data using resources readily
at hand...

Setup:
- OS X host running a tool to measure persist/recover
- VirtualBox host with 2 CPU, 3 GB RAM running Ubuntu 14.04 and MySQL 5.5.58

Network RTT from host to guest OS tends to be between 250µs and 500µs.

The tool calls `persist()` many times, reports stats, then invokes `recover()`.  I explored
3 use cases - minimal `SaveQuota` records, large (100 KB) records, and `persist()` calls that
batch many records.  See results below:
```console
-- Simple records (minimal SaveQuota records)
Persisted 50000 times in 92653 ms (539 persists/sec)
  avg 1.85 ms
  p50 1.27 ms
  p90 2.54 ms
  p99 11.56 ms
Recovered 50000 records in 354 ms (141,242 records/sec)

-- Large records (ScheduledTasks with 100 KB blobs)
Persisted 10000 times in 98392 ms (101 persists/sec)
  avg 9.84 ms
  p50 6.19 ms
  p90 20.95 ms
  p99 36.95 ms
Recovered 10000 records in 9605 ms (1,041 records/sec)

-- Record batches (each persist is 100 simple records)
Persisted 10000 times in 71286 ms (140 persists/sec)
  avg 7.13 ms
  p50 4.32 ms
  p90 14.64 ms
  p99 32.12 ms
Recovered 1000000 records in 25494 ms (39,224 records/sec)
```
This is at best a corse approximation, as server-class hardware should improve these figures;
and there would be a tax if MySQL clustering were in play.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64288/#review192666
-----------------------------------------------------------


On Dec. 3, 2017, 5:09 p.m., Bill Farner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64288/
> -----------------------------------------------------------
> 
> (Updated Dec. 3, 2017, 5:09 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Introduces a `Persistence` implementation that uses a SQL database via JDBC.  I've opted
to lean towards MySQL SQL dialect, as unfortunately there are vendor differences for even
the very simple SQL used here.
> 
> I chose [HikariCP](https://github.com/brettwooldridge/HikariCP) to serve as a `DataSource`
(connection pool) implementation.  We don't really need much out of a connection pool aside
from general connection lifecycle management (i.e. not for concurrency).  I chose this library
based on recent development activity and several positive comparisons to other pools.
> 
> Note that the implementation is not yet wired into the scheduler application.  That will
come in a follow-up.
> 
> 
> Diffs
> -----
> 
>   build.gradle af119910e84c48f75f2573ababcfa287c3b986fc 
>   config/spotbugs/excludeFilter.xml 51790cce8d9047e40741f05ee55af15dbdc3065e 
>   src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorage.java 85b2113631586f43d854c4d2812f43b7b864d452

>   src/main/java/org/apache/aurora/scheduler/storage/durability/Persistence.java 9eb862c01bf451252bfbcc7a2eac60d2c965c9f0

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogPersistence.java e70e6051582ca90ae72014626b983bbf4b8d5b48

>   src/main/java/org/apache/aurora/scheduler/storage/sql/SqlPersistence.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/sql/SqlPersistenceModule.java PRE-CREATION

>   src/main/resources/org/apache/aurora/scheduler/storage/sql/schema.sql PRE-CREATION

>   src/test/java/org/apache/aurora/scheduler/storage/sql/SqlPersistenceTest.java PRE-CREATION

> 
> 
> Diff: https://reviews.apache.org/r/64288/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Bill Farner
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message