couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Amies <damie...@gmail.com>
Subject Re: CouchDB / NoSQL Benchmarking
Date Wed, 04 Nov 2015 13:14:51 GMT
2. Performance degradation due to being Memory bound
3. Performance degradation due to being CPU bound

These 2 scenarios happened on the same test, again the benchmark script and
CouchDB were running on the same machine. This machine is a Early 2015
Macbook Pro (i5 and 8GB Ram), quite a reasonable machine, however it had a
lot of other programs running (3 browsers with many tabs, 2 VirtualBox
Virtual machines, code editors, etc) so before starting the benchmark the
amount of ram free was less than 0.8GB. once the disk cache used up the
available ram and the machine became memory bound, once this happened the
machine quickly became CPU bound as well. the benchmark continued to run in
this state until completion. producing a poor result as expected, but also
helping to proving the robustness of CouchDB to continue running in this
state for sustained periods of time.

The system info:
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
  Model Name: MacBook Pro
  Model Identifier: MacBookPro12,1
  Processor Name: Intel Core i5
  Processor Speed: 2.9 GHz
  Number of Processors: 1
  Total Number of Cores: 2
  L2 Cache (per Core): 256 KB
  L3 Cache: 3 MB
  Memory: 8 GB
  Boot ROM Version: MBP121.0167.B07
  SMC Version (system): 2.28f7
------------------------------------------------------------------
------------------------------------------------------------------
----------------------


and the benchmark test output:
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
    DavesMBP:KVBench dave$ python2.7 KVBench.py
    Started DataPrime(100) 10/21/15 23:08:39 AEST
    Finished DataPrime(100) 10/21/15 23:18:21 AEST
    DataPrime took 581.941123962 Seconds
    Started Phase 1(50) 10/21/15 23:18:21 AEST

.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    Finished Phase 1(50) 10/21/15 23:36:13 AEST
    Phase 1 took 1071.63343596 Seconds
    Started Phase 2(50) 10/21/15 23:36:13 AEST

.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    Finished Phase 2(50) 10/22/15 00:20:24 AEST
    Phase 2 took 2651.14099789 Seconds
    Started Phase 3(50) 10/22/15 00:20:24 AEST

.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    Finished Phase 3(50) 10/22/15 00:41:24 AEST
    Phase 3 took 1259.85448599 Seconds
    Phase 1
    All : 5.19307534019
    Query 3.1 : 27.0711379695
    Query 3.2 : 63.8858062992
    Query 3.3 : 1.67225912619
    Read : 0.0604927515984
    Write : 0.0553895394802
    Phase 2
    All : 28.4322744925
    Query 3.1 : 16.7036313298
    Query 3.2 : 74.0986278311
    Query 3.3 : 0.0969976773262
    Read : 0.031681224823
    Write : 0.026964152813
    Phase 3
    All : 8.09359945008
    Query 3.1 : 15.2892434311
    Query 3.2 : 97.713498662
    Query 3.3 : 0.0960221848488
    Read : 0.019273829174
    Write : 0.0188897314072
    KV Bench v1.0.3 score: 13.9063164276
------------------------------------------------------------------
------------------------------------------------------------------
----------------------



On Tue, Oct 27, 2015 at 8:18 PM, Garren Smith <garren@apache.org> wrote:

> Hi Dave,
>
> This is very cool. Do you have the results and the scripts you used to
> benchmarch CouchDB?
>
> Cheers
> Garren
>
> On Thu, Oct 22, 2015 at 3:13 PM, Dave Amies <damies13@gmail.com> wrote:
>
> > Hi All,
> >
> > I'm sure by now most of you will have read at least some parts of this
> > guide:
> >
> > http://guide.couchdb.org/draft/performance.html
> >
> > I was reading it the other day and noticed the "Call to Arms" section at
> > the bottom of the page. I don't know if there are already any
> benchmarking
> > tools out there, but I decided to try writing one. Hopefully the one I
> have
> > written will be useful.
> >
> > About my background, for my day job i am a performance tester, usually
> > specialising in Loadrunner, so this project was something to keep my mind
> > occupied while waiting for my test system to be rebuilt. Given this I
> have
> > only spent a few hours on it and so there is probably still room for
> > improvement, this email is about finding out if there is interest or if
> > this will be useful to the CouchDB community, so really should I continue
> > developing this tool, or am I wasting my time?
> >
> > In designing this benchmarking utility I reflected on all the systems I
> > have tested and tried to come up with some common areas where database
> > systems suffer in performance. Then bearing in mind the fundamental
> > differences between traditional databases and NoSQL databases
> (particularly
> > CouchDB) I tried to construct some some common database usage scenarios.
> >
> > The 3 scenarios I came up with are:
> >
> >    1. Write heavy (each user performs 12 writes, 6 reads and 3 searches /
> >    index queries)
> >    2. Index / Query / Search heavy (each user performs 1 write, 2 reads
> and
> >    6 searches / index queries)
> >    3. Read Heavy (each user performs 1 writes, 10 reads and 3 searches /
> >    index queries)
> >
> > I have tried out my benchmarking tool on a couple of machines so far, in
> > these tests I managed to cause CouchDB to encounter the following
> > situations:
> >
> >    1. Performance degradation due to being Disk IO bound
> >    2. Performance degradation due to being Memory bound
> >    3. Performance degradation due to being CPU bound
> >    4. Couch DB crashed
> >    5. Benchmarking completed successfully and produce a performance score
> >
> > Based on these results I believe I have created an effective tool for
> > benchmarking, so I decided the best next step was to release the tool as
> an
> > open source project, so I created a github project which can be found
> here:
> > https://github.com/damies13/kvbench. Here you will the readme file
> > describes the 3 scenarios in more detail, the benchmark definition or
> > design and also the pre benchmark data priming. You will also find here
> the
> > python script that is the benchmarking tool and some instructions for
> > setting up a couch db database for the benchmarking process.
> >
> > As this is getting long i'll wrap up by noting that I deliberately did
> not
> > use the python couchdb libraries but instead I used the requests library
> > (standard http) and json library because I wanted to keep the code as
> > generic as possible, the intention is that this benchmarking tool should
> be
> > able to be used to benchmarking any key / value store, whether that be a
> > document based NoSQL, and Key Value based NoSQL database or some other
> Rest
> > API / engine (e.g. backed by a traditional database).
> >
> > I look forward to some feed back, hopefully I have created something
> > useful.
> >
> > Sincerely,
> >
> > Dave.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message